yt-dlp: [Broken] Oreilly download support not working anymore and the playlist items give 404

Checklist

I’m reporting a broken site support
I’ve verified that I’m running yt-dlp version 2021.07.24
I’ve checked that all provided URLs are alive and playable in a browser
I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
I’ve searched the bugtracker for similar issues including closed ones

Verbose log

D:\Utils\YoutubeDL\DownloadOreilly>python "D:\Utils/YoutubeDL/ytdlsrc/yt-dlp/yt_dlp/__main__.py" --config-location "D:\Utils/YoutubeDL/OreillyConf/youtube-dl.conf" --playlist-start 1 --download-archive archive.txt --cookies cookies.txt
[debug] Custom config file: D:\Utils/YoutubeDL/OreillyConf/youtube-dl.conf
[debug] Custom config: ['-u', 'PRIVATE', '-p', 'PRIVATE', '-2', '-i', '-c', '-v', '--no-warnings', '--console-title', '--batch-file=batch-file.txt', '--write-annotations', '--write-description', '--write-info-json', '--write-thumbnail', '--sub-lang', 'en', '--write-auto-sub', '--write-sub', '--add-metadata', '--embed-thumbnail', '-o', '%(playlist_title)s/%(playlist_index)s. %(title)s.%(ext)s', '-f', 'bestvideo[height<=720]+bestaudio/best[height<=720]/worst', '--merge-output-format', 'mp4', '--mark-watched', '--geo-bypass']
[debug] Command-line config: ['--config-location', 'D:\\Utils/YoutubeDL/OreillyConf/youtube-dl.conf', '--playlist-start', '1', '--download-archive', 'archive.txt', '--cookies', 'cookies.txt']
[debug] Batch file urls: ['https://learning.oreilly.com/videos/learning-path-delivering/9781491989012/']
[debug] Loading archive file 'archive.txt'

[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.24 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: 11cc45718
[debug] Python version 3.9.1 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-4-gacb339bb88, ffprobe n4.4-4-gacb339bb88
[debug] Proxy map: {}
[safari:course] Downloading login page
[debug] [safari:course] Extracting URL: https://learning.oreilly.com/videos/learning-path-delivering/9781491989012/
[safari:course] 9781491989012: Downloading course JSON
[download] Downloading playlist: Learning Path: Delivering Applications with Docker
[info] Writing playlist metadata as JSON to: Learning Path - Delivering Applications with Docker\0. Learning Path - Delivering Applications with Docker.info.json
[safari:course] playlist Learning Path: Delivering Applications with Docker: Collected 74 videos; downloading 74 of them
[download] Downloading video 1 of 74
[safari:api] Downloading login page
[debug] [safari:api] Extracting URL: https://learning.oreilly.com/api/v1/book/9781491989012/chapter/video301824.html
[safari:api] 9781491989012/video301824: Downloading part JSON
[safari] Downloading login page
[debug] [safari] Extracting URL: https://learning.oreilly.com/library/view/learning-path-delivering/9781491989012/video301824.html
[safari] 9781491989012-video301824: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
  File "D:\Utils\YoutubeDL\ytdlsrc\yt-dlp\yt_dlp\extractor\common.py", line 679, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "D:\Utils\YoutubeDL\ytdlsrc\yt-dlp\yt_dlp\YoutubeDL.py", line 3151, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

These same configs worked just 2 days ago, now I can’t download any Oreilly content, I did try to check the links, they do give 404 from a browser.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 22 (16 by maintainers)

Most upvoted comments

[debug] Custom config: ['--cookies-from-browser', 'firefox', '--download-archive', 'archive.txt', '-i', '-c', '-v', '--console-title', '--batch-file=batch-file.txt', '--write-annotations', '--write-description', '--write-info-json', '--write-thumbnail', '--sub-lang', 'en', '--write-auto-sub', '--write-sub', '--add-metadata', '--embed-subs', '--embed-thumbnail', '-o', '%(playlist_title)s/%(playlist_index)s. %(title)s.%(ext)s', '-f', 'bestvideo[height<=720]+bestaudio/best[height<=720]/worst', '--merge-output-format', 'mp4']
[debug] Command-line config: ['--config-location', 'D:\\Utils/YoutubeDL/Configs/ytdlp_oreilly.conf', '--playlist-items', '2']
[debug] Batch file urls: ['https://learning.oreilly.com/videos/the-principles-of/9781491935811/']
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "C:\Users\mh-sh\AppData\Roaming\Mozilla\Firefox\Profiles\8tceuger.default-release\cookies.sqlite"
[Cookies] Extracted 1548 cookies from firefox
[debug] Loading archive file 'archive.txt'

[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.09.02 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: 982323fe1
[debug] Python version 3.9.1 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-4-gacb339bb88, ffprobe n4.4-4-gacb339bb88
[debug] Optional libraries: sqlite
[debug] Proxy map: {}
[debug] [safari:course] Extracting URL: https://learning.oreilly.com/videos/the-principles-of/9781491935811/
[safari:course] 9781491935811: Downloading course JSON
[download] Downloading playlist: The Principles of Microservices
[info] Writing playlist metadata as JSON to: The Principles of Microservices\0. The Principles of Microservices.info.json
WARNING: There's no playlist description to write.
[safari:course] playlist The Principles of Microservices: Collected 15 videos; downloading 1 of them
[download] Downloading video 1 of 1
[debug] [safari:api] Extracting URL: https://learning.oreilly.com/api/v1/book/9781491935811/chapter/video221406.html
[safari:api] 9781491935811/video221406: Downloading part JSON
[debug] [safari] Extracting URL: https://learning.oreilly.com/videos/the-principles-of/9781491935811/9781491935811-video221406/
[debug] [Kaltura] Extracting URL: https://cdnapisec.kaltura.com/html5/html5lib/v2.37.1/mwEmbedFrame.php?wid=_1926081&uiconf_id=29375172&flashvars%5BreferenceId%5D=9781491935811-video221406
[Kaltura] 9781491935811-video221406: Downloading webpage
[Kaltura] 0_1i6jb4o4: Downloading video info JSON
[Kaltura] 0_1i6jb4o4: Checking mp4-4468 URL
[Kaltura] 0_1i6jb4o4: Downloading m3u8 information
WARNING: [Kaltura] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Downloading subtitles: en
[info] 0_1i6jb4o4: Downloading 1 format(s): mp4-1966
WARNING: There's no description to write.
WARNING: There are no annotations to write.
[info] Writing video subtitles to: The Principles of Microservices\2. What are Microservices.en.ttml
[debug] Invoking downloader on "http://cdnapi.kaltura.com/api_v3/service/caption_captionasset/action/serve/captionAssetId/0_diyhxwtn"
[download] The Principles of Microservices\2. What are Microservices.en.ttml has already been downloaded
[download] 100% of 15.08KiB
[info] Writing video metadata as JSON to: The Principles of Microservices\2. What are Microservices.info.json
[Kaltura] 0_1i6jb4o4: Downloading thumbnail  ...
[Kaltura] 0_1i6jb4o4: Writing thumbnail to: The Principles of Microservices\2. What are Microservices.jpg
[debug] Invoking downloader on "http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/0_1i6jb4o4/format/url/protocol/http/flavorId/0_coc8hy0d"
[download] Resuming download at byte 2717283
[download] Destination: The Principles of Microservices\2. What are Microservices.mp4
[download]   7.6% of 35.87MiB at 489.40KiB/s ETA 01:09

The issue now is that the downloader is invoked on “http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/0_1i6jb4o4/format/url/protocol/http/flavorId/0_coc8hy0d” which is only 3min, but it already had access to the full video at “https://cdnapisec.kaltura.com/html5/html5lib/v2.37.1/mwEmbedFrame.php?wid=_1926081&uiconf_id=29375172&flashvars[referenceId]=9781491935811-video221406”

MKSherbini on Sep 16, 2021

@MKSherbini I tested this with my credentials both with individual video page and course page. It’s downloading the videos without issues

hasantayyar on Sep 17, 2021

I just installed with python3 -m pip install --upgrade git+https://github.com/MKSherbini/yt-dlp and was able to download again with a free trial account.

~~BUT all videos are truncated after 60 seconds.’ When using the same account in their webplayer it’s possible to watch the videos beyond the 60s~~

Update: I think the trunkated videos happened because I still was using an (old) cookie.txt and a useragent option taken from earlier tries to get yt-dlp and safari working. I am sorry for the confusion.

I now can confirm that MKSherbini is working for me, even with a trial account! Thanks MKSherbini!

grepmeister on Sep 18, 2021

@hasantayyar Thanks, I submitted a PR to handle this change here. As I didn’t read the rest of the code I can’t confirm if it creates other issues, but now I can download again without issues.

MKSherbini on Sep 16, 2021