yt-dlp: ZDF.de broken / [ZDFChannel] Unable to extract document id

Checklist

  • I’m reporting that yt-dlp is broken on a supported site
  • I’ve verified that I have updated yt-dlp to nightly or master (update instructions)
  • I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
  • I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
  • I’ve searched known issues, the FAQ, and the bugtracker for similar issues including closed ones. DO NOT post duplicates
  • I’ve read about sharing account credentials and I’m willing to share it if required

Region

Germany

Provide a description that is worded well enough to be understood

ZDF introduced a new design for their website yesterday. Now all downloads fail.

Here are a couple test URLs, all of which fail (I couldn’t find a single example on the site that doesn’t):

Interestingly, if using the old URL format, the downloads still work. These permanently redirect (308) to new URLs, but yt-dlp can download them no problem. Running yt-dlp on the new URLs fails, however. Some examples found through googling – there’s not really a way to reverse-engineer these that I could find:

But for these examples, even merely appending “.html” to the new URLs works, though not with the “staffel” URL and the “heute-19-uhr” category page.

yt-dlp seems to use a different extractor for those, showing “[ZDF]” on the old ones (working) and “[ZDFChannel]” on the new ones (not working), so it seems to miscategorize the new URL format.

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['https://www.zdf.de/video/magazine/heute-journal-104/heute-journal-vom-16-maerz-2025-100', '--skip-download', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds [2ee3a0aff] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.26100-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 7.1-essentials_build-www.gyan.dev (setts), ffprobe 7.1-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.22.0, brotli-1.1.0, certifi-2025.01.31, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.3.0, websockets-15.0.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Plugin directories: none
[debug] Loaded 1847 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds)
[debug] Using fake IP 53.140.162.95 (DE) as X-Forwarded-For
[ZDFChannel] Extracting URL: https://www.zdf.de/video/magazine/heute-journal-104/heute-journal-vom-16-maerz-2025-100
[ZDFChannel] heute-journal-vom-16-maerz-2025-100: Downloading webpage
ERROR: [ZDFChannel] heute-journal-vom-16-maerz-2025-100: Unable to extract document id; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 747, in extract
  File "yt_dlp\extractor\zdf.py", line 487, in _real_extract
  File "yt_dlp\extractor\common.py", line 1338, in _search_regex

[debug] Command-line config: ['https://www.zdf.de/nachrichten-sendungen/heute-journal/heute-journal-vom-16-maerz-2025-100.html', '--skip-download', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds [2ee3a0aff] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.26100-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 7.1-essentials_build-www.gyan.dev (setts), ffprobe 7.1-essentials_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.22.0, brotli-1.1.0, certifi-2025.01.31, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.3.0, websockets-15.0.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Plugin directories: none
[debug] Loaded 1847 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2025.03.16.232921 from yt-dlp/yt-dlp-nightly-builds)
[debug] Using fake IP 53.147.186.36 (DE) as X-Forwarded-For
[ZDF] Extracting URL: https://www.zdf.de/nachrichten-sendungen/heute-journal/heute-journal-vom-16-maerz-2025-100.html
[ZDF] heute-journal-vom-16-maerz-2025-100: Downloading webpage
[ZDF] heute-journal-vom-16-maerz-2025-100: Downloading JSON metadata
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[ZDF] 250316_2145_sendung_hjo: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec, channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 250316_2145_sendung_hjo: Downloading 1 format(s): http-h264_aac_mp4_http_na_na-fhd

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 24
  • Comments: 24 (8 by maintainers)

Commits related to this issue

Most upvoted comments

I’ve dug through the API requests a bit more. Turns out this is actually easier than expected, if we just model the extractor structure after the CMS structure that they have. In particular: treat everything that is not a /video or /play URL as collection/playlist. Here is my current idea of an approach:

ZDFChannelIE

_VALID_URL = r'https?://www\.zdf\.de/(?:[^/?#]+/)*(?P<id>[^/?#]+)'

@classmethod
def suitable(cls, url):
    return False if ZDFIE.suitable(url) else super().suitable(url)

def _real_extract(self, url):
    channel_id = self._match_id(url)
    webpage, urlh = self._download_webpage_handle(url, channel_id)
    # Make sure to get the correct ID in case of redirects
    channel_id = self._search_regex(self._VALID_URL, urlh.url, 'channel id',  group='id')
    collection_data = self._download_graphql(channel_id, 'downloading smart collection data', {
        'operationName': 'GetSmartCollectionByCanonical',
        'variables': f'{{"canonical":"{channel_id}","videoPageSize":24}}',
        'extensions': '{"persistedQuery":{"version":1,"sha256Hash":"cb49420e133bd668ad895a8cea0e65cba6aa11ac1cacb02341ff5cf32a17cd02"}}',
    })

collection_data contains a bunch of JSON.

  • If traverse_obj(collection_data, ('data', 'smartCollectionByCanonical', 'video') exists, return a URL result with video['sharingUrl']. This will be a URL starting with https://www.zdf.de/video/ and be handled by ZDFIE.
  • If traverse_obj(collection_data, ('data', 'smartCollectionByCanonical', 'seasons') exists, generate playlist result from seasons.
  • If both exist, maybe give a choice through _yes_playlist(). Though I haven’t seen that case yet.

ZDFIE

_VALID_URL = [
    r'https?://(?:www\.)?zdf\.de/(?:video|play)/(?:[^/?#]+/)*(?P<id>[^/?#]+)/?',
    # Legacy URLs that redirect
    r'https?://(?:www\.)?zdf\.de/(?:[^/?#]+/)*(?P<id>[^/?#]+)\.html',
]

def _real_extract(self, url):
    video_id = self._match_id(url)
    webpage = self._download_webpage(url, video_id)
    video_data = self._download_graphql(video_id, 'downloading video metadata', {
        'operationName': 'GetVideoMetaByCanonical',
        'variables': f'{{"canonical":"{video_id}"}}',
        'extensions': '{"persistedQuery":{"version":1,"sha256Hash":"737eb4421d274259baa3051929f4ecfef2d2afc59f12a9d82285c14dbdd1dd0d"}}',
    })

Then it is simply traverse_obj(video_data, ('data', 'videoByCanonical', 'currentMedia', 'nodes', 0, 'ptmdTemplate')) which can be used to call the existing _extract_ptmd().


I guess I’ll take a stab at implementing that then.

OK, an open PR to fix this issue is up now at #12779. This is quiet an extensive change. So if anyone who has the repo cloned wants to test it, that would be appreciated. I have tested pretty extensively I think, but there is always a chance I’ve missed something.

Thanks for looking into this.

Apparently, the format for programme pages is

And the format for episode pages is

A different season’s URL is always the programme URL with “?staffel=[season]” appended.

So it’s not specifically with “magazine”. There can be many different categories.

Some examples:

programme: https://www.zdf.de/shows/bares-fuer-rares-104 different season: https://www.zdf.de/shows/bares-fuer-rares-104?staffel=2022 episode: https://www.zdf.de/video/shows/bares-fuer-rares-104/bares-fuer-rares-vom-18-maerz-2025-100

programme: https://www.zdf.de/serien/the-rookie-100 different season: https://www.zdf.de/serien/the-rookie-100?staffel=5 episode: https://www.zdf.de/video/serien/the-rookie-100/der-prozess-110

programme: https://www.zdf.de/reportagen/die-spur-224 different season: https://www.zdf.de/reportagen/die-spur-224?staffel=2023 episode: https://www.zdf.de/video/reportagen/die-spur-224/finanzen-influencer-geldanlage-socialmedia-boerse-copytrading-100

programme: https://www.zdf.de/dokus/faszination-erde-mit-hannah-emde-dokureihe-100 different season: https://www.zdf.de/dokus/faszination-erde-mit-hannah-emde-dokureihe-100?staffel=19 episode: https://www.zdf.de/video/dokus/faszination-erde-mit-hannah-emde-dokureihe-100/faszination-erde-ueberleben-planet-im-wandel-mit-dirk-steffens-100

programme: https://www.zdf.de/kurzfassungen/bundesliga-fussball-highlights-zusammenfassung-100 different season: https://www.zdf.de/kurzfassungen/bundesliga-fussball-highlights-zusammenfassung-100?staffel=19 episode: https://www.zdf.de/play/kurzfassungen/bundesliga-fussball-highlights-zusammenfassung-100/vfl-wolfsburg-holstein-kiel-highlights-zusammenfassung-saison-2024-25-100

Movies and standalone programmes function like programmes with a single episode:

main page: https://www.zdf.de/filme/a-breath-away-movie-100 player: https://www.zdf.de/play/filme/a-breath-away-movie-100/a-breath-away-100

main page: https://www.zdf.de/konzerte/judas-priest-epitaph-movie-100 player: https://www.zdf.de/play/konzerte/judas-priest-epitaph-movie-100/judas-priest-epitaph-live-at-hammersmith-apollo-100

Workaround:

Open Networktab in FF play video stop video check networktab for AKAMAI URL ending with .webm yt-dlp url Rename manually

The patch is not complete. The ZDF extractor implements two different APIs: The web API at api.zdf.de and the Android API at zdf-prod-futura.zdf.de. Extraction via web API is silently skipped because the extractor doesn’t find the necessary configuration data in the changed page structure anymore. So now it falls back to the Android API which only requires the document ID form the URL and doesn’t look at the page content at all. But for the fallback Android API less metadata is extracted than for the web API. Also looking at network traffic, there is a (possibly new) GraphQL endpoint at api.zdf.de/graphql as well now.

So there is a couple options for this:

  • Find the necessary information again and repair extraction via web API. Though it may be that this API will be replaced with the GraphQL API and get removed entirely.
  • Extend Android API extraction to match the information from the web API and make it the primary extraction. Though the web API would likely still be required for the 3sat extractor, for which the Android API did not work last time I tried.
  • Look into the GraphQL API and see if that can be the primary API for extraction.
  • There is also a .m3u8 URL available inside a script tag. That may be another option, but it provides only a smaller set of formats and metadata would need to be extracted in a different way.

@InvalidUsernameException

So if anyone who has the repo cloned wants to test it, that would be appreciated

I tested and my problem/URL from zdf.de was solved 👍🏼

🦎 ✔ ~/dev/github/yt-dlp-fix/bundle/docker/static [fix-zdf-after-website-rewrite|✔]
12:05 $ docker compose up --build static
…
🦎 ✔ ~/dev/github/yt-dlp-fix/bundle/docker/static [fix-zdf-after-website-rewrite|✚ 1]
12:08 $ docker compose up -d
…
🦎 thomas@merz-nimbus:/automnt/external_2tb [3/2245]
12:09 $ docker exec -it docker-static-1 /yt-dlp/yt-dlp.sh https://www.zdf.de/play/dokus/terra-x-harald-lesch-102/und-wie-ki-uns-ueberwacht-100
ERROR: Invalid update channel '' requested. Valid channels are stable, nightly, master
[ZDF] Extracting URL: https://www.zdf.de/play/dokus/terra-x-harald-lesch-102/und-wie-ki-uns-ueberwacht-100
[ZDF] und-wie-ki-uns-ueberwacht-100: Downloading API token
[ZDF] und-wie-ki-uns-ueberwacht-100: Downloading video metadata
[ZDF] und-wie-ki-uns-ueberwacht-100: Downloading PTMD data
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] und-wie-ki-uns-ueberwacht-100: Downloading PTMD data
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[ZDF] 240917_sendung_thl_dgs: Downloading m3u8 information
[info] und-wie-ki-uns-ueberwacht-100: Downloading 1 format(s): http-h264_aac_mp4_http_na_na-fhd-2
[download] Destination: ... und wie KI uns überwacht [und-wie-ki-uns-ueberwacht-100].mp4
[download]   0.4% of    1.10GiB at    2.74MiB/s ETA 06:48

A workaround seems to be --force-generic

So I don’t think there is any reliable way to tell whether a page is a single video or a playlist just based on the URL alone. We would need to download at least the webpage do make that determination. I assume doing a network request in suitable() would be a big no-no.

Yeah

Does that mean that ZDFIE and ZDFChannelIE need to be combined into a single extractor?

Not necessarily. I think we could do something like this:

First, in ZDFIE:

    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        # The code to determine whether the page is for a video or for a playlist goes here.
        # It should assign `playlist_id` a value other than `None` only if the page is for a playlist
        if self._yes_playlist(playlist_id, video_id):
            return self.url_result(smuggle_url(url, {'is_playlist': True}))

and then in ZDFChannelIE:

    @classmethod
    def suitable(cls, url):
        from ..utils import unsmuggle_url

        url, smuggled_data = unsmuggle_url(url, {})
        return False if not smuggled_data.get('is_playlist') else super().suitable(url)

and the ZDFChannelIE entries should probably have 'force_noplaylist': True smuggled in their URL results just to be safe (unless the site has nested playlists, which I really really hope they don’t)

Hmm, this gets more annoying the closer you look at it. As shown by the various examples already posted, we can’t reliably determine whether a certain URL is a playlist or an individual video based on number of path segments. We know https://www.zdf.de/video/ and https://www.zdf.de/play/ will be individual files, but https://www.zdf.de/[category]/[id] might both be an individual video or a playlist.

From the examples it looked like a specific category would contain either only video or only playlists. But that is not true either. There are for example the following:

So I don’t think there is any reliable way to tell whether a page is a single video or a playlist just based on the URL alone. We would need to download at least the webpage do make that determination. I assume doing a network request in suitable() would be a big no-no. Does that mean that ZDFIE end ZDFChannelIE need to be combined into a single extractor?


I’ve also dug a little bit into the Android app again. That forced me to update to the latest version before proceeding. Looking at the traffic after the update, I’ve found that it has stopped using what I previously referred to as the “Android API” (i.e. zdf-prod-futura.zdf.de) for most of its requests. Instead it now also uses the GraphQL API for nearly all requests.

Combined with the findings from dirkf about discrepancies between what is shown on the website and what the API returns, this does not sound good. It might be that the CMS data behind the old API is no longer updated for things that aired since the redesign or similar.

Presumably that means that zdf-prod-futura.zdf.de is mostly deprecated and the extractor should stop relying on it.

I think the way forward here is using the GraphQL API. At a cursory look, it seems like it will contain everything we need for extraction, both for single items and for playlists. At least once we have determined which page is which to be able to make the correct requests.


The same URL works past 23:00 Europe/Berlin, so the issue due to the rating-based time constraint.

@theMarix That is weird, the API URL in use (https://zdf-prod-futura.zdf.de/mediathekV2/document/fight-club-movie-100) did not contain any data for me, even during the correct time yesterday.

Again, there is no sign of any network API being used to populate a single-video page, although the GraphQL API operation getNotificationsForBookmarks is used – is this a geo-restriction issue?

@dirkf Seems likely. I do get a GraphQL request once I click play on a video page. But I don’t get any GraphQL requests when the video is age-restricted and I press play outside the allowed time of day. So presumably you can’t see them because the geo restriction error works similarly.

The old URLs succeed because they match the extractor pattern, where the ID is extracted, and then the redirected page has the same structure despite the new URL.

ZDFChannel only matches if ZDF fails, so fixing the ZDF extractor pattern solves that.

The heute-19-uhr page is a playlist with a lead video; similarly for /serien/the-tourist-110?staffel=1, which is a playlist of “season” 1 of The Tourist with episode 1 as lead video. Fortunately ZDFChannel (almost) knows how to handle these, with --no-playlist selecting the lead video:

  • we have to assume that /magazine/ URLs have at least 1 further path component ahead of the ID
  • some hacking is needed to get season 1 instead of season 2 for the Tourist page, and to avoid trailer episodes
  • finally some episodes fail with an empty result from _download_v2_doc(), but then the whole show is geo-restricted from the UK: the website says “Dieser Inhalt ist in deinem Land nicht verfügbar”.

Tests have to be added and exercised (patch updated per https://github.com/yt-dlp/yt-dlp/issues/12647#issuecomment-2735103753):

--- old/yt-dlp/yt_dlp/extractor/zdf.py
+++ new/yt-dlp/yt_dlp/extractor/zdf.py
@@ -10,6 +10,7 @@
     join_nonempty,
     merge_dicts,
     parse_codecs,
+    parse_qs,
     qualities,
     traverse_obj,
     try_get,
@@ -249,7 +250,10 @@
 
 
 class ZDFIE(ZDFBaseIE):
-    _VALID_URL = r'https?://www\.zdf\.de/(?:[^/]+/)*(?P<id>[^/?#&]+)\.html'
+    _VALID_URL = (
+        r'https?://www\.zdf\.de/(?:play/)?(?:video|(?:magazine|reportagen|konzerte|filme)/[^/?#&]+)(?:/[^/?#&]+)*/(?P<id>[^/?#&.]+)',
+        r'https?://www\.zdf\.de/(?:[^/]+/)*(?P<id>[^/?#&]+)\.html',
+    )
     _TESTS = [{
         # Same as https://www.phoenix.de/sendungen/ereignisse/corona-nachgehakt/wohin-fuehrt-der-protest-in-der-pandemie-a-2050630.html
         'url': 'https://www.zdf.de/politik/phoenix-sendungen/wohin-fuehrt-der-protest-in-der-pandemie-100.html',
@@ -471,26 +475,50 @@
                 'duration': ('length', {float_or_none}),
                 'season_number': ('seasonNumber', {int_or_none}),
                 'episode_number': ('episodeNumber', {int_or_none}),
+                'timestamp': ('visibleFrom', {unified_timestamp}),
             }))
 
     def _entries(self, data, document_id):
+        structure_node_path = traverse_obj(data, ('document', 'structureNodePath'))
+        if structure_node_path:
+            def on_brand(v):
+                return v['structureNodePath'] == structure_node_path
+        else:
+            def on_brand(v):
+                return v['brandId'] == document_id
+
         for entry in traverse_obj(data, (
             'cluster', lambda _, v: v['type'] == 'teaser',
+            'teaser', lambda _, v: v['type'] == 'video' and v.get('contentType') in ('episode', None)
             # If 'brandId' differs, it is a 'You might also like' video. Filter these out
-            'teaser', lambda _, v: v['type'] == 'video' and v['brandId'] == document_id and v['sharingUrl'],
+            and on_brand(v) and v['sharingUrl'],
         )):
             yield self._extract_entry(entry)
 
     def _real_extract(self, url):
         channel_id = self._match_id(url)
         webpage = self._download_webpage(url, channel_id)
+        hero_id = self._search_json(
+            re.escape(r'\"heroVideo\":'), webpage, 'hero video', channel_id,
+            transform_source=lambda s: s.encode().decode('unicode_escape'), default={})
+        hero_id = hero_id.get('canonical')
+        if hero_id:
+            return self._extract_mobile(hero_id)
+
         document_id = self._search_regex(
-            r'docId\s*:\s*(["\'])(?P<doc_id>(?:(?!\1).)+)\1', webpage, 'document id', group='doc_id')
+            r'docId\s*:\s*(["\'])(?P<doc_id>(?:(?!\1).)+)\1', webpage, 'document id', group='doc_id',
+            default=channel_id)
         data = self._download_v2_doc(document_id)
-
+        teaser_content = {'teaserContent'}
+        season_id = parse_qs(url).get('staffel', [None])[-1]
+        if season_id:
+            data['cluster'] = traverse_obj(data, ('cluster', lambda _, v: re.match(rf'Staffel\s+{season_id}(?:\W|$)', v['name'])))
+            teaser_content.add('teaser')
         main_video = traverse_obj(data, (
-            'cluster', lambda _, v: v['type'] == 'teaserContent',
-            'teaser', lambda _, v: v['type'] == 'video' and v['basename'] and v['sharingUrl'], any)) or {}
+            'cluster', lambda _, v: v['type'] in teaser_content,
+            'teaser', lambda _, v: (
+                v['type'] == 'video' and v['basename'] and v['sharingUrl']
+                and v.get('contentType') in ('episode', None)), any)) or {}
 
         if not self._yes_playlist(channel_id, main_video.get('basename')):
             return self._extract_entry(main_video)