yt-dlp: [TubiTV] JSONDecodeError -- detect geo-block and add `--geo-verification-proxy` support

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

I’m reporting that yt-dlp is broken on a supported site
I’ve verified that I’m running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
I’ve read the guidelines for opening an issue
I’ve read about sharing account credentials and I’m willing to share it if required

Region

America

Provide a description that is worded well enough to be understood

When I use yt-dlp -F tubitvlink I get this following error. Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); this started happing only on 2022-09-03.

Provide verbose output that clearly demonstrates the problem

Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
If using API, add 'verbose': True to YoutubeDL params instead
Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

—REMOVED—

About this issue

Original URL
State: open
Created 10 months ago
Comments: 33 (15 by maintainers)

Most upvoted comments

Adding the geo-verification-proxy support only has an effect if you specify a proxy. I guess --no-geo-bypass is what’s meant above? Obviously the XFF tactic fails for this site and should be disabled.

I mean it currently defaults to this: [debug] Using fake IP 6.10.104.246 (US) as X-Forwarded-For [TubiTv] Extracting URL: https://tubitv.com/tv-shows/351987/s01-e01-blues

Possibly because of this? _GEO_COUNTRIES = ['US']

If I try to download a show that’s available in Australia but not US, it will fail. It also doesn’t allow me to download US shows so it’s useless. I have to use --no-geo-bypass to fix the issue.

I don’t see /oz/ anywhere in the urls.

Jules-A on Sep 5, 2023

I’ve verified that the tokens aren’t specific for each video.

The tokens are specific to country, though:

"country":"US"

so swapping an AU token into a US manifest URL might not do any good

bashonly on Sep 8, 2023

The JWT is in the manifest link. And the JWT expires:

"exp":1694274900

you can’t change the exp time because of the above-mentioned signature

bashonly on Sep 7, 2023

The JWTs are signed. Unless the signing key can be found in client-side JS (which is rare, but known to happen), it’s impossible

bashonly on Sep 7, 2023

Do all geo-blocked regions get the same page, or does the page vary by region and/or language? Eg, do non-English-speaking EU countries get a localised version?

My EU, non-English speaking 😜, country gets the https://gdpr.tubi.tv/ page in English, when trying to access https://tubitv.com/movies/501355

tubi1

The same redirection takes place when trying to access directly the json “page” https://tubitv.com/oz/videos/501355/content …

If a show is known but not available in the current region, is there a useful indication of that in the JSON result?

From a US IP address (via HTTPS proxy):

https://tubitv.com/oz/videos/501355/content

yields this; quoting:

      "availability_starts": "2018-11-07T00:00:00.000Z",
      "availability_ends": null,

OTOH, from an AU IP address (via HTTPS proxy):

https://tubitv.com/oz/videos/501355/content

yields this, with

      "availability_starts": "2023-09-07T21:38:26.989Z",
      "availability_ends": "2024-03-07T21:38:26.989Z",

i.e. it’ll become available in Down Under imminently and shall remain available there for half a year…

Obviously, I can’t see a way for an American TubiTV user to tell that the same title is or isn’t available in Oz, or, conversely, for an Aussie TubiTV user to tell that the same title isn’t or is available (at the same time) in the States 😉 … But I could be wrong, what do I know? 😄

Vangelis66 on Sep 7, 2023

Does --xff US work in Australia

No. The extractor does this by default, and as Jules-A said above:

If I try to download a show that’s available in Australia but not US, it will fail. It also doesn’t allow me to download US shows so it’s useless. I have to use --no-geo-bypass to fix the issue.

bashonly on Sep 7, 2023

Also, https://en.wikipedia.org/w/index.php?title=Tubi#Global_availability. The article as a whole doesn’t support an Australian origin theory.

Not sure what you are trying to say? It didn’t launch with Australia support but added on September 1, 2019.

Jules-A on Sep 6, 2023

Also, https://en.wikipedia.org/w/index.php?title=Tubi#Global_availability. The article as a whole doesn’t support an Australian origin theory.

dirkf on Sep 6, 2023

Or maybe the site devs were just following the yellow brick road …

dirkf on Sep 6, 2023

For /oz/, see https://github.com/yt-dlp/yt-dlp/issues/8018#issuecomment-1704310119, or the extractor source; otherwise, you confirmed what I thought.

dirkf on Sep 5, 2023

That could explain this in the JSON API URL:

(/oz/ ??)

Adding the geo-verification-proxy support only has an effect if you specify a proxy. I guess --no-geo-bypass is what’s meant above? Obviously the XFF tactic fails for this site and should be disabled.

dirkf on Sep 5, 2023

The site says it’s only available in the USA, but there’s no geo-restriction test in the extractor (same in yt-dl).

It’s also available in Australia but has significantly less titles. Last time I tried the extractor over a year ago the extractor failed because of the forced US geo proxy causing 403s if I could recall. I had to disable the proxy to get it to work… I haven’t checked the code but does this take in to account that Australians can also watch it and don’t want a forced US IP?

Jules-A on Sep 5, 2023

Can you please check if that currently works for that site?

It does not

Surely support for that option doesn’t have to be added to each extractor, does it?

Yes, it does. self.geo_verification_headers() needs to be added to the headers for the request(s) where proxy is needed

bashonly on Sep 5, 2023

So, the extractor is eligible for a --geo-verification-proxy patch…

Can you please check if that currently works for that site? Surely support for that option doesn’t have to be added to each extractor, does it?

gamer191 on Sep 5, 2023

Can’t repro from US IP:

$ yt-dlp -F "https://tubitv.com/movies/697559/sherlock-homes-the-case-of-the-christmas-pudding"
[TubiTv] Extracting URL: https://tubitv.com/movies/697559/sherlock-homes-the-case-of-the-christmas-pudding
[TubiTv] 697559: Downloading JSON metadata
[TubiTv] 697559: Downloading MPD manifest
[TubiTv] 697559: Downloading m3u8 information
[TubiTv] 697559: Downloading m3u8 information
[info] Available formats for 697559:
ID                                 EXT RESOLUTION │   FILESIZE   TBR PROTO │ VCODEC        VBR ACODEC     ABR ASR MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
hlsv6-default-audio-group-stream_1 mp4 audio only │                  m3u8  │ audio only        unknown            [en] stream_1
dash-0                             m4a audio only │ ~ 12.74MiB   66k https │ audio only        mp4a.40.2  66k 44k [en] DASH audio, m4a_dash
dash-2                             mp4 426x312    │ ~ 53.31MiB  275k https │ avc1.4d401e  275k video only         DASH video, mp4_dash
hlsv6-345                          mp4 426x312    │ ~ 66.99MiB  345k m3u8  │ avc1.4d401e  345k video only
hlsv3-321                          mp4 426x312    │ ~ 62.26MiB  321k m3u8  │ avc1.4D401E       mp4a.40.2
dash-3                             mp4 640x470    │ ~126.28MiB  651k https │ avc1.4d401e  651k video only         DASH video, mp4_dash
hlsv6-709                          mp4 640x470    │ ~137.70MiB  710k m3u8  │ avc1.4d401e  710k video only
hlsv3-695                          mp4 640x470    │ ~134.81MiB  695k m3u8  │ avc1.4D401E       mp4a.40.2
dash-4                             mp4 854x626    │ ~255.08MiB 1315k https │ avc1.640028 1315k video only         DASH video, mp4_dash
hlsv6-1354                         mp4 854x626    │ ~262.68MiB 1354k m3u8  │ avc1.640028 1354k video only
hlsv3-1420                         mp4 854x626    │ ~275.44MiB 1420k m3u8  │ avc1.640028       mp4a.40.2

bashonly on Sep 3, 2023