yt-dlp: [pbs] An extractor error has occurred. (caused by KeyError('title'))

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Region

New York, United States

Provide a description that is worded well enough to be understood

Example URL: https://www.pbs.org/wgbh/nova/video/decoding-the-universe-cosmos/

This error occurs after the “Downloading portalplayer page” regardless of yt-dlp arguments. Tested on both git master (with Python 3.11.10 in a venv) and 2024.11.04 (with Python 3.12.7), both on macOS 10.12. The debug output below is from git master.

This is anonymous viewing; I do not have credentials to share.

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://www.pbs.org/wgbh/nova/video/decoding-the-universe-cosmos/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.11.04 from yt-dlp/yt-dlp [197d0b03b]
[debug] Lazy loading extractors is disabled
[debug] Python 3.11.10 (CPython x86_64 64bit) - macOS-10.12.6-x86_64-i386-64bit (OpenSSL 3.4.0 22 Oct 2024)
[debug] exe versions: ffmpeg 4.4.5 (fdk,setts), ffprobe 4.4.5, rtmpdump 2.4
[debug] Optional libraries: sqlite3-3.47.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib
[debug] Loaded 1838 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.11.04 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.11.04 from yt-dlp/yt-dlp)
[debug] Using fake IP 6.163.93.244 (US) as X-Forwarded-For
[pbs] Downloading JSON metadata
[pbs] Extracting URL: https://www.pbs.org/wgbh/nova/video/decoding-the-universe-cosmos/
[pbs] decoding-the-universe-cosmos: Downloading webpage
[pbs] Downloading widget/partnerplayer page
[pbs] Downloading portalplayer page
ERROR: An extractor error has occurred. (caused by KeyError('title')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/private/tmp/xytdlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 742, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/xytdlp/lib/python3.11/site-packages/yt_dlp/extractor/pbs.py", line 689, in _real_extract
    'title': info['title'],
             ~~~~^^^^^^^^^
KeyError: 'title'

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 45 (16 by maintainers)

Commits related to this issue

Most upvoted comments

Is that something that can be fixed, given time?

Yes, but the entire PBS extractor will need to be re-assessed in light of the PBS site redesign, and we may want/need to devise a general solution for the new version of NextJS data. It could take a while

This seems to have been fixed. I can now download NOVA episodes again.

Yes it has been fixed. It was part of the 2024.01.26 release (see the release notes for PBS fix).

The existing pattern handles video.pbssocal.org/xxxx/... for certain words xxxx as well as www.pbs.org/whatever.... PR #9740 adds support for www.pbssocal.org/whatever... as a special case, but it might be better to generalise that for other stations:

...
         (r'vids\.kvie\.org', 'KVIE Public Television (KVIE)'),  # http://www.kvie.org
-        (r'video\.pbssocal\.org', 'PBS SoCal/KOCE (KOCE)'),  # http://www.pbssocal.org/
+        (r'(?:video\.|www\.)pbssocal\.org', 'PBS SoCal/KOCE (KOCE)'),  # http://www.pbssocal.org/
         (r'video\.valleypbs\.org', 'ValleyPBS (KVPT)'),  # http://www.valleypbs.org/
...
     _VALID_URL = r'''(?x)https?://
         (?:
-           # Direct video URL
-           (?:%s)/(?:(?:vir|port)alplayer|video)/(?P<id>[0-9]+)(?:[?/]|$) |
-           # Article with embedded player (or direct video)
-           (?:www\.)?pbs\.org/(?:[^/]+/){1,5}(?P<presumptive_id>[^/]+?)(?:\.html)?/?(?:$|[?\#]) |
            # Player
-           (?:video|player)\.pbs\.org/(?:widget/)?partnerplayer/(?P<player_id>[^/]+)/
+           (?:video|player)\.pbs\.org/(?:widget/)?partnerplayer/(?P<player_id>[^/]+) |
+           # Direct video URL, or article with embedded player
+           (?:%s)/(?:
+              (?:(?:vir|port)alplayer|video)/(?P<id>[0-9]+)(?:[?/]|$) |
+              (?:[^/]+/){1,5}(?P<presumptive_id>[^/]+?)(?:\.html)?/?(?:$|[?\#])
+           )
         )
-    ''' % '|'.join(list(zip(*_STATIONS))[0])
+    ''' % '|'.join(next(zip(*_STATIONS)))

The 2 patches above by dirkf do fix it for all the most common shows I have tried. Could these be merged in now to get it working until a more comprehensive update is ready later?

Oops, the {1,5} needs to be escaped with double braces as in the last deleted line.

There are two problems with the URLs above:

In both cases the player URL is found in both the #fixed-iframe-text and the #responsive-iframe-text <input> elements as the src of the <iframe> element encoded as the value of each <input>. In the el-gato-feo case, there is also a TVEpisode ld+json block whose .video is a VideoObject containing "contentUrl": "https://player.pbs.org/viralplayer/3092772965".

A search of each original page for r'https?://player\.pbs\.org/[\w-]+player/\d+/' would find the media link with no more complex page parsing.

Maybe PBS members other than RMPBS have yet further structures, or there are pages with false positive player links, but for these URLs:

     _VALID_URL = r'''(?x)https?://
         (?:
-           # Direct video URL
-           (?:{})/(?:(?:vir|port)alplayer|video)/(?P<id>[0-9]+)(?:[?/]|$) |
-           # Article with embedded player (or direct video)
-           (?:www\.)?pbs\.org/(?:[^/]+/){{1,5}}(?P<presumptive_id>[^/]+?)(?:\.html)?/?(?:$|[?\#]) |
+           # Player (match player_id ahead of presumptive_id)
+           (?:video|player)\.pbs\.org/(?:widget/)?partnerplayer/(?P<player_id>[^/]+) |
+           # Direct video URL, or article with embedded player
+           (?:{})/(?:
+              (?:(?:vir|port)alplayer|video)/(?P<id>[0-9]+)(?:[?/]|$) |
+              (?:[^/]+/){{1,5}}(?P<presumptive_id>[^/]+?)(?:\.html)?/?(?:$|[?\#])
+           )
-           # Player
-           (?:video|player)\.pbs\.org/(?:widget/)?partnerplayer/(?P<player_id>[^/]+)
         )
     '''.format('|'.join(next(zip(*_STATIONS))))

and

             MEDIA_ID_REGEXES = [
+                r'\bhttps?://player\.pbs\.org/[\w-]+player/(\d+)',      # move to end to avoid false +ves?

                 r"div\s*:\s*'videoembed'\s*,\s*mediaid\s*:\s*'(\d+)'",  # frontline video embed
                 r'class="coveplayerid">([^<]+)<',                       # coveplayer
                 r'<section[^>]+data-coveid="(\d+)"',                    # coveplayer from http://www.pbs.org/wgbh/frontline/film/real-csi/
                 r'<input type="hidden" id="pbs_video_id_[0-9]+" value="([0-9]+)"/>',  # jwplayer
                 r"(?s)window\.PBS\.playerConfig\s*=\s*{.*?id\s*:\s*'([0-9]+)',",
                 r'<div[^>]+\bdata-cove-id=["\'](\d+)"',  # http://www.pbs.org/wgbh/roadshow/watch/episode/2105-indianapolis-hour-2/
                 r'<iframe[^>]+\bsrc=["\'](?:https?:)?//video\.pbs\.org/widget/partnerplayer/(\d+)',  # https://www.pbs.org/wgbh/masterpiece/episodes/victoria-s2-e1/
             ]

Changes: bracing escape, matching order

I don’t want to distract too much from the main discourse of this post, but it sounds like a proper fix to the PBS extractor could take a little while. So, for those interested, until then, there are some alternate ways to get at the PBS video information, depending on your skill set and the software at your disposal. I’m an amateur at all of this, so this information may not work for you, and it probably isn’t the best way or the only way of doing this (i.e., the command arguments I’m using may not be optimal). My experience is using Firefox in Windows 10.

Even with the new version of the PBS website, I’ve found so far in my limited tests that when I play a video (my example is from the new Ken Burns Leonardo da Vinci documentary, part 1: “https://www.pbs.org/video/leonardo-da-vinci-part-one-the-disciple-of-experience/”), I can right-click in the video window and get the menu option “This Frame -> View Frame Source”. When I do that, I get HTML information that about halfway down says things like

window.videoBridge = {“availability”: “available”, “encodings”: [“https://urs.pbs.org/redirect/c8b63c97d228496a8dd70e7760934c29/”, “https://urs.pbs.org/redirect/5472e443957446bf95b8defab3019618/”], “has_hls_encodings”: true, “has_mp4_encodings”: true

The first URL in this case is typically the 1080p video with a default audio track that YT-DLP gets that is often Spanish or descriptive English audio, so I’ve been trying to download all available audio tracks and get them merged into 1 file with the 1080p video. The second URL is typically a 720p MP4 video that seems more likely to just have the English language audio already in it. So, in this example, if I were OK with just the 720p quality video and whatever default, hopefully English, audio track, I can run a command like this that uses that second URL and I’m done:

yt-dlp.exe -o 720p_Leonardo_da_Vinci_Part_1.mp4 --hls-prefer-ffmpeg “https://urs.pbs.org/redirect/5472e443957446bf95b8defab3019618/”

This probably gets me the most stable file using my ignorant methods. If I want the higher quality video, knowing that there are often multiple audio tracks and English often isn’t the default (so I try to get all of the audio tracks), I run a command like this that uses the first URL:

yt-dlp.exe -f “bv*+mergeall[vcodec=none]” --audio-multistreams --merge-output-format “mp4/mkv” -o 1080p_Leonardo_da_Vinci_Part_1.mkv --hls-prefer-ffmpeg “https://urs.pbs.org/redirect/c8b63c97d228496a8dd70e7760934c29/”

In this example, the final video has 3 audio tracks: Spanish, Descriptive English, and English. Sometimes this video doesn’t play right in some video players for some reason, but between VLC, Media Player, and the Windows Movies and TV app, I feel like something can usually play it. So, maybe my method doesn’t make the most stable video file. I’m not sure. I’m not an expert on this stuff.

The other way to download the video would be like I think people have had to do on the PBS Kids site where their video player doesn’t seem to allow for this “View Frame Source” approach to the information. Before playing the video, turn on your web browser’s debugging and network monitoring tools to look for “.m3u8” files. In Firefox, I use the “Open Application Menu” menu option (the triple horizontal bar icon), and select “More Tools -> Web Developer Tools”. In the Web Developer Tool, I select the Network Monitor tab and type “.m3u8” into the search filter. Then I start playing the video. The Network Monitor tends to find 3 or 4 files when I do this. In my example, it lists under “File”:

ldvi0101-AABR-AVC_830.m3u8 ldvi0101-AABR-AVC-540p_1296k_4.m3u8 ldvi0101-AABR-AVC_eng_0_VBR_48_225791.m3u8 ldvi0101-AABR-AVC-720p_2794k_3.m3u8

which, if you right-click on them and select “Copy Value -> Copy URL”, gives you these corresponding URLs:

https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_830.m3u8

https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC-540p_1296k_4.m3u8

https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_eng_0_VBR_48_225791.m3u8

https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC-720p_2794k_3.m3u8

The first URL typically has the highest quality 1080p video and multiple audio tracks where English isn’t the default if that third file with “eng” in the name shows up. The second and fourth files (the ones with “540p” and “720p” in the names) are lower-resolution videos that also don’t seem to have any audio at all. The third file (the one with “eng” in the name) has no video, but has the English language track information. So, one can either do like I did above and try to download the 1080p file with all of its audio tracks into 1 file using the first URL from the Network Monitor, like

yt-dlp.exe -f “bv*+mergeall[vcodec=none]” --audio-multistreams --merge-output-format “mp4/mkv” -o 1080p_Try_2_Leonardo_da_Vinci_Part_1.mkv --hls-prefer-ffmpeg “https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_830.m3u8”

or, I do two downloads: one to get the 1080p video with its default (non-English) audio from the first URL, and the second to get just the English audio track from the third URL. I’ve been doing this using these two commands:

yt-dlp.exe -o Leonardo_da_Vinci_Part_1_Sp.mp4 --hls-prefer-ffmpeg “https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_830.m3u8”

yt-dlp.exe -o Leonardo_da_Vinci_Part_1_Eng.mp4 -x --audio-format “aac” --audio-quality 0 --hls-prefer-ffmpeg “https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_eng_0_VBR_48_225791.m3u8”

This gives me the 1080p video + Spanish language file “Leonardo_da_Vinci_Part_1_Sp.mp4” and the English audio file “Leonardo_da_Vinci_Part_1_Eng.m4a”. I then have to use a third-party program (like Avidemux) to add the English audio track to the 1080p video file to make the final video file I want. So, several extra steps, but it’s been working so far.

Hopefully once the PBS extractor is fixed, none of this will be necessary, but the extractor has been working so well for so long, some people may not have realized that there are other manual ways of getting access to the video files, and those options appear to still be available even with the redesigned PBS website at the moment, but things can change at any moment.

I tried the patch above, but didn’t see how it would address this problem, and, of course, it doesn’t.

The problem is that this same error can occur for many different reasons. The title KeyError is just a symptom of the actual video page not being extracted. Of course we can make title extraction non-fatal in a proper PR, but it’s irrelevant while we are debugging the video ID extraction.

The link you provided indeed is not fixed by either of the above patches. It looks like it’s because that page is using the new major version of NextJS where the hydration “JSON” is actually a bunch of push calls in separate script tags, for which our _search_nextjs_data method is ineffective.

Is extraction working for the video from https://github.com/yt-dlp/yt-dlp/issues/11514#issuecomment-2488965467 (Masterpiece Theater, Little Women E3, https://www.pbs.org/video/episode-3-r7iod3/) with the current yt-dlp extractor (and so, without processing the new Next.js scripts)?

works for me

Thanks for getting the fix implemented!

This seems to have been fixed. I can now download NOVA episodes again.

A workaround posted to issue #11928, now closed, by user october262

try again with yt-dlp --force-generic-extractor “https://www.pbs.org/video/take-a-chance-wdZQCx/” the video will now download.

Tried it on one PBS Nova video and it worked.

The PBS extractor error is still present in the latest yt-dlp to stable@2024.12.23 on Windows 11. Will have to see about trying the 2 patches by dirkf above.

yt-dlp -vU https://www.pbs.org/video/lost-tombs-of-notre-dame-ayjw0k/
[debug] Command-line config: ['-vU', 'https://www.pbs.org/video/lost-tombs-of-notre-dame-ayjw0k/']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.12.23 from yt-dlp/yt-dlp [65cf46cdd] (win_exe)
[debug] Python 3.10.11 (CPython AMD64 64bit) - Windows-10-10.0.26100-SP0 (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 7.0.2-full_build-www.gyan.dev (setts), ffprobe 7.0.2-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.21.0, brotli-1.1.0, certifi-2024.12.14, curl_cffi-0.5.10, mutagen-1.47.0, requests-2.32.3, sqlite3-3.40.1, urllib3-2.3.0, websockets-14.1
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1837 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.12.23 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.12.23 from yt-dlp/yt-dlp)
[debug] Using fake IP 6.14.213.138 (US) as X-Forwarded-For
[pbs] Downloading JSON metadata
[pbs] Extracting URL: https://www.pbs.org/video/lost-tombs-of-notre-dame-ayjw0k/
[pbs] lost-tombs-of-notre-dame-ayjw0k: Downloading webpage
[pbs] Downloading widget/partnerplayer page
[pbs] Downloading portalplayer page
ERROR: An extractor error has occurred. (caused by KeyError('title')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 742, in extract
  File "yt_dlp\extractor\pbs.py", line 689, in _real_extract
KeyError: 'title'

Thanks. I looked at it several times and still missed that. 😃

I can confirm that it’s working now for the “www.pbs.org/video” URLs. Thanks!

Thanks, dirkf!

I tried to apply your patches, but I’m getting this:

File "/home/hunter/software/yt-dlp/yt_dlp/extractor/pbs.py", line 196, in PBSIE '''.format('|'.join(next(zip(*_STATIONS)))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyError: '1,5'

And I can’t see why. I’ve checked it several times—my replacement lines are identical to what you’ve posted. Ideas?

Thanks! (I also don’t know why my line breaks aren’t being displayed in my Code inserts.)

@someziggyman no need to open a new issue. We can use this one as a blanket issue for the site redesign; the new Next.js version has already been discussed in here, etc. Don’t want to split info/discussion into multiple threads

Just a FYI, you should use -o "1080p_Leonardo_da_Vinci_Part_1.%(ext)s" so that yt-dlp will choose automatically between MP4 and mkv extension