yt-dlp: ZDF.de broken / [ZDFChannel] Unable to extract document id
Checklist
- I’m reporting that yt-dlp is broken on a supported site
- I’ve verified that I have updated yt-dlp to nightly or master (update instructions)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched known issues, the FAQ, and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read about sharing account credentials and I’m willing to share it if required
Region
Germany
Provide a description that is worded well enough to be understood
ZDF introduced a new design for their website yesterday. Now all downloads fail.
Here are a couple test URLs, all of which fail (I couldn’t find a single example on the site that doesn’t):
- https://www.zdf.de/video/dokus/zdf-info-100/panama-duerre-flucht-und-kriminelle-clans-100
- https://www.zdf.de/magazine/heute-19-uhr-102
- https://www.zdf.de/serien/the-tourist-110?staffel=1
- https://www.zdf.de/video/reportagen/am-puls-100/am-puls-corona-tacke-long-covid-impfen-querdenker-100
- https://www.zdf.de/video/talk/maybrit-illner-128/gute-schulden-schlechte-schulden-hat-merz-sich-verzockt-maybrit-illner-vom-13-maerz-2025-100
- https://www.zdf.de/video/shows/zdf-magazin-royale-102/zdf-magazin-royale-vom-14-maerz-2025-100
Interestingly, if using the old URL format, the downloads still work. These permanently redirect (308) to new URLs, but yt-dlp can download them no problem. Running yt-dlp on the new URLs fails, however. Some examples found through googling – there’s not really a way to reverse-engineer these that I could find:
- https://www.zdf.de/nachrichten-sendungen/heute-journal/heute-journal-vom-16-maerz-2025-100.html ✅ → https://www.zdf.de/video/magazine/heute-journal-104/heute-journal-vom-16-maerz-2025-100 ❌
- https://www.zdf.de/verbraucher/wiso/wiso-vom-9-dezember-2024-100.html ✅ → https://www.zdf.de/video/magazine/wiso-104/wiso-vom-9-dezember-2024-100 ❌
- https://www.zdf.de/serien/the-rookie/rueckschlag-100.html ✅ → https://www.zdf.de/video/serien/the-rookie-100/rueckschlag-100
But for these examples, even merely appending “.html” to the new URLs works, though not with the “staffel” URL and the “heute-19-uhr” category page.
yt-dlp seems to use a different extractor for those, showing “[ZDF]” on the old ones (working) and “[ZDFChannel]” on the new ones (not working), so it seems to miscategorize the new URL format.
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>) - If using API, add
'verbose': TruetoYoutubeDLparams instead - Copy the WHOLE output (starting with
[debug] Command-line config) and insert it below
Complete Verbose Output
[debug] Command-line config: ['https://www.zdf.de/video/magazine/heute-journal-104/heute-journal-vom-16-maerz-2025-100', '--skip-download', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds [2ee3a0aff] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.26100-SP0 (OpenSSL 1.1.1t 7 Feb 2023)
[debug] exe versions: ffmpeg 7.1-essentials_build-www.gyan.dev (setts), ffprobe 7.1-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.22.0, brotli-1.1.0, certifi-2025.01.31, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.3.0, websockets-15.0.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Plugin directories: none
[debug] Loaded 1847 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds)
[debug] Using fake IP 53.140.162.95 (DE) as X-Forwarded-For
[ZDFChannel] Extracting URL: https://www.zdf.de/video/magazine/heute-journal-104/heute-journal-vom-16-maerz-2025-100
[ZDFChannel] heute-journal-vom-16-maerz-2025-100: Downloading webpage
ERROR: [ZDFChannel] heute-journal-vom-16-maerz-2025-100: Unable to extract document id; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U
File "yt_dlp\extractor\common.py", line 747, in extract
File "yt_dlp\extractor\zdf.py", line 487, in _real_extract
File "yt_dlp\extractor\common.py", line 1338, in _search_regex
[debug] Command-line config: ['https://www.zdf.de/nachrichten-sendungen/heute-journal/heute-journal-vom-16-maerz-2025-100.html', '--skip-download', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds [2ee3a0aff] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.26100-SP0 (OpenSSL 1.1.1t 7 Feb 2023)
[debug] exe versions: ffmpeg 7.1-essentials_build-www.gyan.dev (setts), ffprobe 7.1-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.22.0, brotli-1.1.0, certifi-2025.01.31, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.3.0, websockets-15.0.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Plugin directories: none
[debug] Loaded 1847 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds)
[debug] Using fake IP 53.147.186.36 (DE) as X-Forwarded-For
[ZDF] Extracting URL: https://www.zdf.de/nachrichten-sendungen/heute-journal/heute-journal-vom-16-maerz-2025-100.html
[ZDF] heute-journal-vom-16-maerz-2025-100: Downloading webpage
[ZDF] heute-journal-vom-16-maerz-2025-100: Downloading JSON metadata
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec, channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 250316_2145_sendung_hjo: Downloading 1 format(s): http-h264_aac_mp4_http_na_na-fhd
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 24
- Comments: 24 (8 by maintainers)
Commits related to this issue
- This applies the patch from yt-dlp/yt-dlp#12647 — committed to theMarix/yt-dlp by theMarix 2 years ago
- [ie/zdf] Fix extractors (#12779) Closes #12647 Authored by: InvalidUsernameException, bashonly Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com> — committed to karavan/yt-dlp by InvalidUsernameException 2 years ago
I’ve dug through the API requests a bit more. Turns out this is actually easier than expected, if we just model the extractor structure after the CMS structure that they have. In particular: treat everything that is not a
/videoor/playURL as collection/playlist. Here is my current idea of an approach:ZDFChannelIEcollection_datacontains a bunch of JSON.traverse_obj(collection_data, ('data', 'smartCollectionByCanonical', 'video')exists, return a URL result withvideo['sharingUrl']. This will be a URL starting withhttps://www.zdf.de/video/and be handled byZDFIE.traverse_obj(collection_data, ('data', 'smartCollectionByCanonical', 'seasons')exists, generate playlist result fromseasons._yes_playlist(). Though I haven’t seen that case yet.ZDFIEThen it is simply
traverse_obj(video_data, ('data', 'videoByCanonical', 'currentMedia', 'nodes', 0, 'ptmdTemplate'))which can be used to call the existing_extract_ptmd().I guess I’ll take a stab at implementing that then.
OK, an open PR to fix this issue is up now at #12779. This is quiet an extensive change. So if anyone who has the repo cloned wants to test it, that would be appreciated. I have tested pretty extensively I think, but there is always a chance I’ve missed something.
Thanks for looking into this.
Apparently, the format for programme pages is
And the format for episode pages is
A different season’s URL is always the programme URL with “?staffel=[season]” appended.
So it’s not specifically with “magazine”. There can be many different categories.
Some examples:
programme: https://www.zdf.de/shows/bares-fuer-rares-104 different season: https://www.zdf.de/shows/bares-fuer-rares-104?staffel=2022 episode: https://www.zdf.de/video/shows/bares-fuer-rares-104/bares-fuer-rares-vom-18-maerz-2025-100
programme: https://www.zdf.de/serien/the-rookie-100 different season: https://www.zdf.de/serien/the-rookie-100?staffel=5 episode: https://www.zdf.de/video/serien/the-rookie-100/der-prozess-110
programme: https://www.zdf.de/reportagen/die-spur-224 different season: https://www.zdf.de/reportagen/die-spur-224?staffel=2023 episode: https://www.zdf.de/video/reportagen/die-spur-224/finanzen-influencer-geldanlage-socialmedia-boerse-copytrading-100
programme: https://www.zdf.de/dokus/faszination-erde-mit-hannah-emde-dokureihe-100 different season: https://www.zdf.de/dokus/faszination-erde-mit-hannah-emde-dokureihe-100?staffel=19 episode: https://www.zdf.de/video/dokus/faszination-erde-mit-hannah-emde-dokureihe-100/faszination-erde-ueberleben-planet-im-wandel-mit-dirk-steffens-100
programme: https://www.zdf.de/kurzfassungen/bundesliga-fussball-highlights-zusammenfassung-100 different season: https://www.zdf.de/kurzfassungen/bundesliga-fussball-highlights-zusammenfassung-100?staffel=19 episode: https://www.zdf.de/play/kurzfassungen/bundesliga-fussball-highlights-zusammenfassung-100/vfl-wolfsburg-holstein-kiel-highlights-zusammenfassung-saison-2024-25-100
Movies and standalone programmes function like programmes with a single episode:
main page: https://www.zdf.de/filme/a-breath-away-movie-100 player: https://www.zdf.de/play/filme/a-breath-away-movie-100/a-breath-away-100
main page: https://www.zdf.de/konzerte/judas-priest-epitaph-movie-100 player: https://www.zdf.de/play/konzerte/judas-priest-epitaph-movie-100/judas-priest-epitaph-live-at-hammersmith-apollo-100
Workaround:
Open Networktab in FF play video stop video check networktab for AKAMAI URL ending with .webm yt-dlp url Rename manually
The patch is not complete. The
ZDFextractor implements two different APIs: The web API atapi.zdf.deand the Android API atzdf-prod-futura.zdf.de. Extraction via web API is silently skipped because the extractor doesn’t find the necessary configuration data in the changed page structure anymore. So now it falls back to the Android API which only requires the document ID form the URL and doesn’t look at the page content at all. But for the fallback Android API less metadata is extracted than for the web API. Also looking at network traffic, there is a (possibly new) GraphQL endpoint atapi.zdf.de/graphqlas well now.So there is a couple options for this:
.m3u8URL available inside a script tag. That may be another option, but it provides only a smaller set of formats and metadata would need to be extracted in a different way.@InvalidUsernameException
I tested and my problem/URL from zdf.de was solved 👍🏼
A workaround seems to be
--force-genericYeah
Not necessarily. I think we could do something like this:
First, in
ZDFIE:and then in
ZDFChannelIE:and the
ZDFChannelIEentries should probably have'force_noplaylist': Truesmuggled in their URL results just to be safe (unless the site has nested playlists, which I really really hope they don’t)Hmm, this gets more annoying the closer you look at it. As shown by the various examples already posted, we can’t reliably determine whether a certain URL is a playlist or an individual video based on number of path segments. We know
https://www.zdf.de/video/andhttps://www.zdf.de/play/will be individual files, buthttps://www.zdf.de/[category]/[id]might both be an individual video or a playlist.From the examples it looked like a specific category would contain either only video or only playlists. But that is not true either. There are for example the following:
So I don’t think there is any reliable way to tell whether a page is a single video or a playlist just based on the URL alone. We would need to download at least the webpage do make that determination. I assume doing a network request in
suitable()would be a big no-no. Does that mean thatZDFIEendZDFChannelIEneed to be combined into a single extractor?I’ve also dug a little bit into the Android app again. That forced me to update to the latest version before proceeding. Looking at the traffic after the update, I’ve found that it has stopped using what I previously referred to as the “Android API” (i.e.
zdf-prod-futura.zdf.de) for most of its requests. Instead it now also uses the GraphQL API for nearly all requests.Combined with the findings from dirkf about discrepancies between what is shown on the website and what the API returns, this does not sound good. It might be that the CMS data behind the old API is no longer updated for things that aired since the redesign or similar.
Presumably that means that
zdf-prod-futura.zdf.deis mostly deprecated and the extractor should stop relying on it.I think the way forward here is using the GraphQL API. At a cursory look, it seems like it will contain everything we need for extraction, both for single items and for playlists. At least once we have determined which page is which to be able to make the correct requests.
@theMarix That is weird, the API URL in use (https://zdf-prod-futura.zdf.de/mediathekV2/document/fight-club-movie-100) did not contain any data for me, even during the correct time yesterday.
@dirkf Seems likely. I do get a GraphQL request once I click play on a video page. But I don’t get any GraphQL requests when the video is age-restricted and I press play outside the allowed time of day. So presumably you can’t see them because the geo restriction error works similarly.
The old URLs succeed because they match the extractor pattern, where the ID is extracted, and then the redirected page has the same structure despite the new URL.
ZDFChannelonly matches ifZDFfails, so fixing theZDFextractor pattern solves that.The
heute-19-uhrpage is a playlist with a lead video; similarly for/serien/the-tourist-110?staffel=1, which is a playlist of “season” 1 of The Tourist with episode 1 as lead video. FortunatelyZDFChannel(almost) knows how to handle these, with--no-playlistselecting the lead video:/magazine/URLs have at least 1 further path component ahead of the ID_download_v2_doc(), but then the whole show is geo-restricted from the UK: the website says “Dieser Inhalt ist in deinem Land nicht verfügbar”.Tests have to be added and exercised (patch updated per https://github.com/yt-dlp/yt-dlp/issues/12647#issuecomment-2735103753):