youtube-dl: Unable to download from ITV Hub

I performed some additional tests with youtube-dl.exe and it looks like Chrome/89 is the minimum Chrome version (at this time 😉 …) that www.itv.com will sanction when requested by yt-dl:

youtube-dl --user-agent "Chrome/88" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"

exhibits the issue reported by OP, while

youtube-dl --user-agent "Chrome/89" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"

successfully gets past the [ITV] 1a7045a0288: Downloading webpage stage and on to [ITV] 1a7045a0288: Downloading JSON metadata; actually, setting the Chrome version to a big, future, value also does the trick (e.g. --user-agent "Chrome/200"); JFYI… 😃

I’m not completely au fait with the filesystem of a Mac, but the file you’re after can be found on the ‘Files changed’ tab of #30266 - here’s a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py

You’ll need to find out where the /youtube_dl/extractor/ folder is on your machine, and download the file overwriting the existing version. Probably best to take a copy of the original before you alter it. You’ll probably be able to see the path somewhere in the output if you add the --verbose switch when you run the command.

Hope that helps.

Perfect - installed that version of itv.py and it worked a treat - thank you

Today, using #30266 and --user-agent 'Chrome/89':

$ python -m youtube_dl -F -v 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89' 
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'-v', u'https://www.itv.com/hub/celebrity-juice/1a7045a0288', u'--user-agent', u'Chrome/89']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Git HEAD: 0945174fa
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP 25.39.215.163 (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[info] Available formats for 1a7045a0288:
format code  extension  resolution note
hls-136      mp4        audio only  136k , mp4a.40.2@128k
hls-318      mp4        512x288     318k , avc1.4D401E@ 171k, 50.0fps, mp4a.40.2@128k
hls-601      mp4        512x288     601k , avc1.4D401F@ 438k, 50.0fps, mp4a.40.2@128k
hls-802      mp4        512x288     802k , avc1.4D401F@ 628k, 50.0fps, mp4a.40.2@128k
hls-1082     mp4        896x504    1082k , avc1.4D401F@ 892k, 50.0fps, mp4a.40.2@128k
hls-1670     mp4        896x504    1670k , avc1.4D401F@1447k, 50.0fps, mp4a.40.2@128k (best)
$ python -m youtube_dl --test 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89' 
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 340
[download] Destination: Celebrity Juice - Series 25 - Episode 1-1a7045a0288.mp4
[download] 100% of 10.00KiB in 00:00
$

OT TV critic: see how I used the --test option to avoid getting a viewable version of the show itself! The selection of Plebs and Wossie as test shows in the extractor was more acceptable.

Apologies but I don’t fully understand the above - should this be working now using --user-agent “Chrome/89” ?

When using the example above, I get ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: ‘Not Found’>);

I am running youtube-dl version 2021.12.17

The betting is that ITV has employed a bouncer that rejects or ignores your web requests if it doesn’t like the look of them.

For the moment, --user-agent 'Mozilla/5.0' confuses the bouncer into allowing access, like some suede trainers.

In your log, yt-dl sent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36 as if it was Chrome 70 on Windows 10 64-bit, but perhaps the bouncer only accepts that UA with HTTP/2 (Chrome 51 on).

… The link points to https://github.com/ytdl-org/youtube-dl/issues/url 😉 ; no doubt a mishap 😄 , but it should be fixed…

@Vangelis66 - fixed - nice spot 😉

ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.

Have you tried the --hls-prefer-native switch? youtube-dl defaults to using ffmpeg for dumping AES-128 encrypted HLS streams (it’s the type of mobile streams the ITV Info Extractor “harvests” in order to fetch the ITVHub programme 😜 ), but said switch will dump the stream via a pure Python implementation, no external downloader needed… However, since this is on a Mac, you may have to install an additional “decrypting” python module (hlsnative dependency), such as the deprecated PyCrypro or the active PyCryprodome equivalent; consult some Mac documentation on how to install one of these modules… But, as your log shows, ffmpeg is still required to mux downloaded stream to the MP4 container, so it’s a good thing you installed it anyway… 😃

… I did notice was the quality wasn’t particularly great, 896x504 however this is probably a limitation of ITV hub?

Yes; look at the formats tables posted by me here and by dirkf here (resolution column) …

Cheers 😄

here’s a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py

… The link points to https://github.com/ytdl-org/youtube-dl/issues/url 😉 ; no doubt a mishap 😄 , but it should be fixed…

I expect your current desktop browser is sending the slightly different HTTP headers.