yt-dlp: [pbs] An extractor error has occurred. (caused by KeyError('title'))
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I intentionally remove or skip any mandatory* field
Checklist
- I’m reporting that yt-dlp is broken on a supported site
- I’ve verified that I have updated yt-dlp to nightly or master (update instructions)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
- I’ve read about sharing account credentials and I’m willing to share it if required
Region
New York, United States
Provide a description that is worded well enough to be understood
Example URL: https://www.pbs.org/wgbh/nova/video/decoding-the-universe-cosmos/
This error occurs after the “Downloading portalplayer page” regardless of yt-dlp arguments. Tested on both git master (with Python 3.11.10 in a venv) and 2024.11.04 (with Python 3.12.7), both on macOS 10.12. The debug output below is from git master.
This is anonymous viewing; I do not have credentials to share.
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>) - If using API, add
'verbose': TruetoYoutubeDLparams instead - Copy the WHOLE output (starting with
[debug] Command-line config) and insert it below
Complete Verbose Output
[debug] Command-line config: ['-vU', 'https://www.pbs.org/wgbh/nova/video/decoding-the-universe-cosmos/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.11.04 from yt-dlp/yt-dlp [197d0b03b]
[debug] Lazy loading extractors is disabled
[debug] Python 3.11.10 (CPython x86_64 64bit) - macOS-10.12.6-x86_64-i386-64bit (OpenSSL 3.4.0 22 Oct 2024)
[debug] exe versions: ffmpeg 4.4.5 (fdk,setts), ffprobe 4.4.5, rtmpdump 2.4
[debug] Optional libraries: sqlite3-3.47.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib
[debug] Loaded 1838 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.11.04 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.11.04 from yt-dlp/yt-dlp)
[debug] Using fake IP 6.163.93.244 (US) as X-Forwarded-For
[pbs] Downloading JSON metadata
[pbs] Extracting URL: https://www.pbs.org/wgbh/nova/video/decoding-the-universe-cosmos/
[pbs] decoding-the-universe-cosmos: Downloading webpage
[pbs] Downloading widget/partnerplayer page
[pbs] Downloading portalplayer page
ERROR: An extractor error has occurred. (caused by KeyError('title')); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U
File "/private/tmp/xytdlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 742, in extract
ie_result = self._real_extract(url)
^^^^^^^^^^^^^^^^^^^^^^^
File "/private/tmp/xytdlp/lib/python3.11/site-packages/yt_dlp/extractor/pbs.py", line 689, in _real_extract
'title': info['title'],
~~~~^^^^^^^^^
KeyError: 'title'
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 45 (16 by maintainers)
Commits related to this issue
- [ie/pbs] Fix extractor (#12024) Closes #8703, Closes #9740, Closes #11514 Authored by: dirkf, krandor, n10dollar Co-authored-by: dirkf <fieldhouse@gmx.net> Co-authored-by: Neil <ntendolkar@berke... — committed to norwd/yt-dlp by krandor 2 years ago
- [ie/pbs] Fix extractor (#12024) Closes #8703, Closes #9740, Closes #11514 Authored by: dirkf, krandor, n10dollar Co-authored-by: dirkf <fieldhouse@gmx.net> Co-authored-by: Neil <ntendolkar@berke... — committed to vladisslav2011/yt-dlp by krandor 2 years ago
- [ie/pbs] Fix extractor (#12024) Closes #8703, Closes #9740, Closes #11514 Authored by: dirkf, krandor, n10dollar Co-authored-by: dirkf <fieldhouse@gmx.net> Co-authored-by: Neil <ntendolkar@berke... — committed to subrat-lima/yt-dlp by krandor 2 years ago
Yes, but the entire PBS extractor will need to be re-assessed in light of the PBS site redesign, and we may want/need to devise a general solution for the new version of NextJS data. It could take a while
Yes it has been fixed. It was part of the 2024.01.26 release (see the release notes for PBS fix).
The existing pattern handles
video.pbssocal.org/xxxx/...for certain wordsxxxxas well aswww.pbs.org/whatever.... PR #9740 adds support forwww.pbssocal.org/whatever...as a special case, but it might be better to generalise that for other stations:The 2 patches above by dirkf do fix it for all the most common shows I have tried. Could these be merged in now to get it working until a more comprehensive update is ready later?
Oops, the
{1,5}needs to be escaped with double braces as in the last deleted line.There are two problems with the URLs above:
video/: the first two sub-patterns in the_VALID_URLneed to be merged;https://player.pbs.org/viralplayer/nnnnnnnnnnin its webpage, but not found by the existingMEDIA_ID_REGEXES.In both cases the player URL is found in both the
#fixed-iframe-textand the#responsive-iframe-text<input>elements as thesrcof the<iframe>element encoded as thevalueof each<input>. In the el-gato-feo case, there is also aTVEpisodeld+json block whose.videois aVideoObjectcontaining"contentUrl": "https://player.pbs.org/viralplayer/3092772965".A search of each original page for
r'https?://player\.pbs\.org/[\w-]+player/\d+/'would find the media link with no more complex page parsing.Maybe PBS members other than RMPBS have yet further structures, or there are pages with false positive player links, but for these URLs:
and
Changes: bracing escape, matching order
I don’t want to distract too much from the main discourse of this post, but it sounds like a proper fix to the PBS extractor could take a little while. So, for those interested, until then, there are some alternate ways to get at the PBS video information, depending on your skill set and the software at your disposal. I’m an amateur at all of this, so this information may not work for you, and it probably isn’t the best way or the only way of doing this (i.e., the command arguments I’m using may not be optimal). My experience is using Firefox in Windows 10.
Even with the new version of the PBS website, I’ve found so far in my limited tests that when I play a video (my example is from the new Ken Burns Leonardo da Vinci documentary, part 1: “https://www.pbs.org/video/leonardo-da-vinci-part-one-the-disciple-of-experience/”), I can right-click in the video window and get the menu option “This Frame -> View Frame Source”. When I do that, I get HTML information that about halfway down says things like
window.videoBridge = {“availability”: “available”, “encodings”: [“https://urs.pbs.org/redirect/c8b63c97d228496a8dd70e7760934c29/”, “https://urs.pbs.org/redirect/5472e443957446bf95b8defab3019618/”], “has_hls_encodings”: true, “has_mp4_encodings”: true
The first URL in this case is typically the 1080p video with a default audio track that YT-DLP gets that is often Spanish or descriptive English audio, so I’ve been trying to download all available audio tracks and get them merged into 1 file with the 1080p video. The second URL is typically a 720p MP4 video that seems more likely to just have the English language audio already in it. So, in this example, if I were OK with just the 720p quality video and whatever default, hopefully English, audio track, I can run a command like this that uses that second URL and I’m done:
yt-dlp.exe -o 720p_Leonardo_da_Vinci_Part_1.mp4 --hls-prefer-ffmpeg “https://urs.pbs.org/redirect/5472e443957446bf95b8defab3019618/”
This probably gets me the most stable file using my ignorant methods. If I want the higher quality video, knowing that there are often multiple audio tracks and English often isn’t the default (so I try to get all of the audio tracks), I run a command like this that uses the first URL:
yt-dlp.exe -f “bv*+mergeall[vcodec=none]” --audio-multistreams --merge-output-format “mp4/mkv” -o 1080p_Leonardo_da_Vinci_Part_1.mkv --hls-prefer-ffmpeg “https://urs.pbs.org/redirect/c8b63c97d228496a8dd70e7760934c29/”
In this example, the final video has 3 audio tracks: Spanish, Descriptive English, and English. Sometimes this video doesn’t play right in some video players for some reason, but between VLC, Media Player, and the Windows Movies and TV app, I feel like something can usually play it. So, maybe my method doesn’t make the most stable video file. I’m not sure. I’m not an expert on this stuff.
The other way to download the video would be like I think people have had to do on the PBS Kids site where their video player doesn’t seem to allow for this “View Frame Source” approach to the information. Before playing the video, turn on your web browser’s debugging and network monitoring tools to look for “.m3u8” files. In Firefox, I use the “Open Application Menu” menu option (the triple horizontal bar icon), and select “More Tools -> Web Developer Tools”. In the Web Developer Tool, I select the Network Monitor tab and type “.m3u8” into the search filter. Then I start playing the video. The Network Monitor tends to find 3 or 4 files when I do this. In my example, it lists under “File”:
ldvi0101-AABR-AVC_830.m3u8 ldvi0101-AABR-AVC-540p_1296k_4.m3u8 ldvi0101-AABR-AVC_eng_0_VBR_48_225791.m3u8 ldvi0101-AABR-AVC-720p_2794k_3.m3u8
which, if you right-click on them and select “Copy Value -> Copy URL”, gives you these corresponding URLs:
https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_830.m3u8
https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC-540p_1296k_4.m3u8
https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_eng_0_VBR_48_225791.m3u8
https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC-720p_2794k_3.m3u8
The first URL typically has the highest quality 1080p video and multiple audio tracks where English isn’t the default if that third file with “eng” in the name shows up. The second and fourth files (the ones with “540p” and “720p” in the names) are lower-resolution videos that also don’t seem to have any audio at all. The third file (the one with “eng” in the name) has no video, but has the English language track information. So, one can either do like I did above and try to download the 1080p file with all of its audio tracks into 1 file using the first URL from the Network Monitor, like
yt-dlp.exe -f “bv*+mergeall[vcodec=none]” --audio-multistreams --merge-output-format “mp4/mkv” -o 1080p_Try_2_Leonardo_da_Vinci_Part_1.mkv --hls-prefer-ffmpeg “https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_830.m3u8”
or, I do two downloads: one to get the 1080p video with its default (non-English) audio from the first URL, and the second to get just the English audio track from the third URL. I’ve been doing this using these two commands:
yt-dlp.exe -o Leonardo_da_Vinci_Part_1_Sp.mp4 --hls-prefer-ffmpeg “https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_830.m3u8”
yt-dlp.exe -o Leonardo_da_Vinci_Part_1_Eng.mp4 -x --audio-format “aac” --audio-quality 0 --hls-prefer-ffmpeg “https://ga.pbs-video.pbs.org/videos/leonardo-da-vinci/4a7af196-0f08-4c00-9a7f-a31dbb69caa6/2000452707/hd-16x9-mezzanine-1080p/ldvi0101-AABR-AVC_eng_0_VBR_48_225791.m3u8”
This gives me the 1080p video + Spanish language file “Leonardo_da_Vinci_Part_1_Sp.mp4” and the English audio file “Leonardo_da_Vinci_Part_1_Eng.m4a”. I then have to use a third-party program (like Avidemux) to add the English audio track to the 1080p video file to make the final video file I want. So, several extra steps, but it’s been working so far.
Hopefully once the PBS extractor is fixed, none of this will be necessary, but the extractor has been working so well for so long, some people may not have realized that there are other manual ways of getting access to the video files, and those options appear to still be available even with the redesigned PBS website at the moment, but things can change at any moment.
The problem is that this same error can occur for many different reasons. The title KeyError is just a symptom of the actual video page not being extracted. Of course we can make
titleextraction non-fatal in a proper PR, but it’s irrelevant while we are debugging the video ID extraction.The link you provided indeed is not fixed by either of the above patches. It looks like it’s because that page is using the new major version of NextJS where the hydration “JSON” is actually a bunch of
pushcalls in separate script tags, for which our_search_nextjs_datamethod is ineffective.works for me
Thanks for getting the fix implemented!
This seems to have been fixed. I can now download NOVA episodes again.
A workaround posted to issue #11928, now closed, by user october262
Tried it on one PBS Nova video and it worked.
The PBS extractor error is still present in the latest yt-dlp to stable@2024.12.23 on Windows 11. Will have to see about trying the 2 patches by dirkf above.
Thanks. I looked at it several times and still missed that. 😃
I can confirm that it’s working now for the “www.pbs.org/video” URLs. Thanks!
Thanks, dirkf!
I tried to apply your patches, but I’m getting this:
File "/home/hunter/software/yt-dlp/yt_dlp/extractor/pbs.py", line 196, in PBSIE '''.format('|'.join(next(zip(*_STATIONS)))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyError: '1,5'And I can’t see why. I’ve checked it several times—my replacement lines are identical to what you’ve posted. Ideas?
Thanks! (I also don’t know why my line breaks aren’t being displayed in my Code inserts.)
@someziggyman no need to open a new issue. We can use this one as a blanket issue for the site redesign; the new Next.js version has already been discussed in here, etc. Don’t want to split info/discussion into multiple threads
Just a FYI, you should use
-o "1080p_Leonardo_da_Vinci_Part_1.%(ext)s"so that yt-dlp will choose automatically between MP4 and mkv extension