youtube-dl: ITV Hub JSON error
Checklist
- I’m reporting a broken site support
- I’ve verified that I’m running youtube-dl version 2021.04.26
- I’ve checked that all provided URLs are alive and playable in a browser
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched the bugtracker for similar issues including closed ones
Verbose log
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.itv.com/hub/tipping-point/2a1875a1123', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[debug] Using fake IP 25.161.160.85 (GB) as X-Forwarded-For.
[ITV] 2a1875a1123: Downloading webpage
[ITV] 2a1875a1123: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\YoutubeDL.py", line 2279, in urlopen
File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
Description
Several programmes on ITV Hub now fail to download. URL’s such as
https://www.itv.com/hub/the-chase/1a7842a1761 https://www.itv.com/hub/tipping-point/2a1875a1123 https://www.itv.com/hub/tenable/10a0805a0021
fail to download though some like
https://www.itv.com/hub/lorraine/1a9360a2830
continue to work
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 47 (4 by maintainers)
Commits related to this issue
- [build] Build Windows x86 version with py3.8 and remove redundant tests :ci skip Ao-authored by: pukkandan, shirt-dev — committed to yt-dlp/yt-dlp by pukkandan 3 years ago
- Merge commit '4c57dd270eb96c717331ed2abf13a3df9644f64c' into v1 * commit '4c57dd270eb96c717331ed2abf13a3df9644f64c': [itv] fixed extraction (closes #28906) — committed to gaming-hacker/youtube-dl by gaming-hacker 3 years ago
- [ITV] Fix extractor, add subtitles and thumbnails (#913) Original PR: https://github.com/ytdl-org/youtube-dl/pull/28955 (see also https://github.com/ytdl-org/youtube-dl/issues/28906#issuecomment-8310... — committed to yt-dlp/yt-dlp by coletdjnz 3 years ago
- [ITV] Fix extractor, add subtitles and thumbnails (#913) Original PR: https://github.com/ytdl-org/youtube-dl/pull/28955 (see also https://github.com/ytdl-org/youtube-dl/issues/28906#issuecomment-8310... — committed to nixxo/yt-dlp by coletdjnz 3 years ago
- [itv] fixed extraction (closes #28906) ITV changes require media stream and subtitles to be taken from different playlist resources. Original commit 4c57dd2 by @sleaux-meaux 3 May 2021 — committed to dirkf/youtube-dl by dirkf 3 years ago
- Merge commit 'c6001d56b2d02d5d470a744f052f49c3a7f7b529' * commit 'c6001d56b2d02d5d470a744f052f49c3a7f7b529': [ITV] Apply default vanilla UA to avoid site blocking Work-around duration parsing pen... — committed to gaming-hacker/youtube-dl by gaming-hacker 2 years ago
@torbs723 wrote:
I was tagged (in a now hidden comment) to troubleshoot this… 😄 It is now already Orthodox Easter Sunday here, I won’t be available for the most part of the next two days… However, since I, too, have a vested interest in ITVHub shows fetched via
yt-dl, I took a stab at it…But first, let’s discuss the
geo-restrictedlabel attributed to this issue… In general, it is not a good idea to aggravate the main developer(s) ofyt-dl😉 ; perhaps the label was just put there to highlight the fact that for anyone to even start troubleshooting this, an ITV Hub whitelisted IP address is required; sometimes, a generic UK IP won’t cut it, you need an “ITV zone” one (e.g. excluding Scotland) … My analysis has indeed proven (to me, at least, but in accordance to comments by other people here) that this new ITV HUB issue is not related to geo-fencing…While I am discussing geo-fencing, the ITV Hub people, some time in the month of March (2021), imposed even stricter geo-blocking schemes… 💢 For many years, their Playlist APIs (= the ones that yield stream data) would only enforce a “soft” geo-block, that could be circumvented via an
X-Forwarded-For:UK IPrequest header; if you look closely at logs people have posted, it’s line[debug] Using fake IP 25.161.160.85 (GB) as X-Forwarded-For.From mid-March, if you happen to be overseas, you need a whitelisted UK HTTPS/SOCKS proxy/UK VPN to even get the master HLS playlist URI (“Downloading JSON metadata” in yt-dl’s logs); of course, the HLS stream itself is fully geo-blocked, you have to use the same “means” to dump if overseas… Again, this new “issue” isn’t related to stricter geo-fencing practices imposed last March (as it also occurs for people within the UK…).It well appears that the ITV Hub people made some changes during these last days, certainly during last week, that make recently uploaded ITV hub shows inaccessible to
yt-dl; case in point is the Jonathan Ross show: Last week’s episodeYerterday’s episode:
(the test was done using the very same connection/IP).
Now, my deeper analysis was done focusing on Viewpoint-SE01EP03 (sadly, the whole series is due to expire imminently): https://www.itv.com/hub/viewpoint/10a0556a0003 As most of you may know already, desktop browsers are being served MPEG-DASH streams with Common Encryption (cenc=DRM), the Google-owned WidevineCDM is required for playback 👎 💢 ; yt-dl can’t/won’t handle cenc, instead it has to resort to HLSe (AES-128 encrypted) streams, targeting older iOS devices…
So the very first thing I did was to emulate such a device in my browser, by setting up a site-specific-user-agent-override (SSUAO) for itv.com; depending on browser used, this can be achieved via an about:config setting (Firefox derived browsers) or, more commonly, via an extension (Chromium browsers) that spoofs your user-agent; please, don’t ask for instructions/how-to here, it isn’t the appropriate place… I chose to use the iPhone5 UA below:
Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3Of course, ITVHub requires you to be logged-in to send any stream data to your browser (this hasn’t been yet a requirement for yt-dl 👍 ); being logged in and with the above “mobile” UA, I inspected browser’s Web Console (CTRL+SHIFT+K) and I clearly saw the POST XHR request to their playlist API:
https://magni.itv.com/playlist/itvonline/ITV/10_0556_0003.001(don’t try this directly with a GET request), the response of which I am attaching here: POST-ITV.json.txt BTW, this is equivalent to yt-dl “Downloading JSON metadata”; as one can see, noHTTP Error 500: Internal Server Errorwas generated in my test; this rules out intermittent server issues on the ITV side…From that (see attached) JSON response, I extracted the master HLSe playlist URI:
(the same URI can be seen and copied from within the browser’s Web Console 😉 ), which looks like below:
This URI is short-lived, you have to act swiftly before its auth-token expires… Feeding that directly to yt-dl, I can initiate a successful download:
youtube-dl --console-title --hls-prefer-native --hls-use-mpegts -c --fixup never "https://itvpnpmobile.content.itv.com/10-0556-0003-001/6/2/VAR003/10-0556-0003-001_6_2_VAR003.ism/.m3u8?hdnea=st=1619890241~exp=1619911841~acl=/10-0556-0003-001/*~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e746976a5d2c26aa99" -o "ITV_Viewpoint_Se1Ep3_202104282100[10a0556a0003].ts"=>I did spend a lot of time trying to emulate the above POST XHR my browser sent in a yt-dl command, by specifying all needed request headers (referer/user-agent/cookies etc.), I even tried the cookies.txt method, sadly all to no avail 😭 … It’s a real pity that I can’t code (in Python or other language), I hope my findings can be put to good use by actual coders…
In conclusion: ITV staff have implemented in recent days “changes” (I wouldn’t rule out this having been done deliberately to thwart downloaders) that would render current itvIE useless once old shows expire, perhaps sooner if these changes propagate to all ITVhub content… At the time of this writing, this isn’t related to an implementation of pure DRM (e.g. Apple FairPlay), nor to further changes to their geo-fencing practices… Therefore, I humbly concluded it’s a genuine itv.py bug that needs to be addressed in code… @remitamine, you seem to be the defacto maintainer of the itvIE, please, when your schedules permit, could you take a deeper look into this? Also pinging @dirkf who happens to be UK-based, a Python coder and quite knowledgeable 😄 in UK media sites…
My best wishes to all, my deepest thanks to the yt-dl dev team 👍 , Happy Easter Sunday to all those that observe it today… 😃
@torbs723: Carrying over from https://github.com/Ascoware/get-iplayer-automator/issues/331#issuecomment-832252389
Use yt-dlp version 2021.12.01 or later
https://github.com/yt-dlp/yt-dlp/releases/latest
@sleaux-meaux : Many thanks 😄 !
For those hesitant to try the linked binary, VirusTotal log below:
(only 1/69 score 👍 )
Preliminary tests have shown it to be working as expected:
All in all, a sterling job 🥇 …
Hello, this is my first time on here asking a question. Firstly I am not into computer programming in any way and find it hard to understand any of the above. I use youtube-dl to download BBC iplayer but had previously used it for ITV until they changed something as stated in the content of this thread. My query is, is there an easy way to download a new version of youtube-dl to allow me to download ITV shows? Is there a very basic guide with step by step instructions at all. Apologies if this is the wrong place to ask. Thanks for any help
Why would you need to apply the “iPhone UA” method TODAY? I merely posted that “method” while at the first stages of me troubleshooting this issue, as a way to detect/verify whether mobile HLSe streams were still being produced by ITVHub infra for the programmes
yt-dlfailed to grab…It is rather unfortunate that the solution to this issue, PR #28955, hasn’t yet been accepted/merged officially (thus, is not present in the officially released binaries 😞 ), but third party
yt-dlbinaries with that “fix” in have been kindly compiled and offered to us 👍 , ITVHub fans 😜, a month ago already, to accommodate:https://github.com/sleaux-meaux/youtube-dl-gia/releases/tag/2021.04.26-gia_05
Were you not able to use one of them? 😲
Excellent work, this method with that exact iPhone UA working today to grab the Euro 2020 Preview show 😃
Considering ITV lost the rights to Pokemon or Champions League highlights a few years back, I would say that page hasn’t been updated in a while.
Great detective work, thanks for investigating this - your prior work/changes were much appreciated too.
No mobile streams, no downloads. That particular show may simply not be available via mobile. Episode noted in https://github.com/ytdl-org/youtube-dl/issues/28910#issuecomment-831177593 is not useful for comparison because it is no longer available, so now produces different 404.
The above (eg https://www.itv.com/hub/ellens-game-of-games/10a0434a0057, S04E09) and the show reported in #28910 are giving 404 rather than 500, so something else is up.
The error occurs when POSTing /playlist/itvonline/ITV2/10_0434_0057.001 to magni.itv.com. This is the
data-video-idattribute of the#videodiv element in the page. The same element listsdata-video-variantswith no variants for themobiletag.The data POSTed is (from l.125 ff. in the extractor):
The
[device][os]data seems to contradict theplatformTag. But 404 implies this was never used.OK, I think I got to the bottom of it… 🎉 My M.O. was comparing the Page Source of ITV Hub shows that work with latest
yt-dl, e.g. (previous) “The Jonathan Ross Show -SE17E03”:https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0204
with the ones that are now “broken”, e.g. (latest) “The Jonathan Ross Show -SE17E04”:
https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205
The point of comparison was the “data-video-variants” block; in the first case (working), it looks like below (pretty-printed, for clarity):
In the second case (non-working shows, giving the “HTTP Error 500”), it looks like below:
As you can see, the ‘dotcom’ group of variants was reduced from 10 to 6 members, the ones that were axed were “softer” encryption methods like (aes | clearkey):
which now leaves us with full-fledged DRM (playready | fairplay | widevine) variants within the ‘dotcom’ group… 😠
As far as
itv.pyitself is concerned,https://github.com/ytdl-org/youtube-dl/blob/a0df8a06178e530a1097f177a1faf1d2c609ac99/youtube_dl/extractor/itv.py#L88-L94
the code above instructs it to request a [‘hls’,‘aes’,‘outband-webvtt’] variant from within the ‘dotcom’ group; this variant is now absent in the non-working shows, the PlaylistAPI server can’t grant
yt-dl’s request, hence theERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Errorthat started this thread…As you can see, ITV Hub are going full-steam ahead with fully DRM-ing all their streams; as a most vocal opponent of DRM in every shape or form, I wish them (…) (fill-in the dots yourself, if you get my drift…).
Workaround:
The only remaining
yt-dl-compatible video-variant left is [‘hls’,‘aes’] inside the ‘mobile’ group; unfortunately, it doesn’t come with the ‘outband-webvtt’ feature that could be found in the now removed variant [‘hls’,‘aes’,‘outband-webvtt’], so subtitles for HLSe streams aren’t accessible 👎 … I have applied locally a quick-and-dirty patch to, at least, regain access to media streams, by doing initv.py:… so I can fetch all these programmes currently broken (bound to increase in number, no doubt 😡 ), but without subs for them, even when subs are provided by ITV:
Obviously, I now need a “real” coder that can harvest the webvtt subtitles from one of the other video-variants (either [‘mpeg-dash’,‘widevine’,‘outband-webvtt’] from the ‘mobile’ group (should be easier?) or from one of the three belonging to the ‘dotcom’ group); any volunteers? 😄
I too have just learnt about spoofing user agents and can concur with @Vangelis66 that downloads from ITV (UK based) are still possible as described above. Happy Easter.
Really appreciate the analysis @Vangelis66. Learnt something about spoofing the user agent as well. Taking it from here is beyond me but hopefuly the coding heroes can find some time and motivation to pick it up. Happy Easter Sunday.
@Vangelis66 As always Vangelis, thank you so much for your detailed analysis of this issue. And thank you for your confirmation that this doesn’t relate to DRM.
If only people would spend 30 seconds doing a search before they post a new issue.
@gewyvoso Thank you for the support
It is possible though that the streams we are having problems with are used by other devices possibly old old iPhones, iPads etc that maybe don’t support ITV’s app?
@gewyvoso I can confirm you get the http 500 error from your first link when I try yt-dl on OSX. I can also confirm that I can access the link from the itv hub with an old iphone using ios. In both cases using a VPN. Is this the info that you want?
I cannot see a 500 error being DRM related. The error shows before the download has even been attempted. If DRM made the videos impossible to download then they would fail with some sort of unable to download error not a server error.
I agree that many programs on ITV Hub now fail to download.
On 28th April I was able to download URL’s
https://www.itv.com/hub/viewpoint/10a0556a0001 https://www.itv.com/hub/viewpoint/10a0556a0002
On 29th April I was unable to download URL
https://www.itv.com/hub/viewpoint/10a0556a0003
Moreover, if I now try to download the first two above URL’s they now fail with the same error message that others are reporting, namely:
So ITV have changed something server side. I don’t think it’s DRM (yet), but maybe a missing header or auth token?
this comment applies to the two issues here and here too, can someone please see whether physical old ios devices are failing on these videos? if so and if itv are still supporting them then can be reported as a bug
500 Internal Server Errors are usually temporary. It means that for some reason, the server was unable to process the request that was sent to it.
If it was DRM related, I would not expect a 500 error.
Someone suggested in the other thread that if you kept trying, the programme would eventually download. That works for 403 Forbidden errors but doesn’t appear to work for this.
The problem seems to be that for some programmes the server is just finding it impossible to send the JSON playlist back which means that YT-DL has no way to attempt to download the programme. Why this only seems to affect some programmes and not others though is a bit of a mystery.
Luckily at the moment, STV player still works, but anything from ITV2, 3, 4 or CITV would not be downloadable because STV is only the Channel 3 contractor for Scotland. And you have to hope they haven’t opted out of the ITV schedule and are showing the show you want.