youtube-dl: oreilly login page error

Checklist

[ x ] I’m reporting a broken site support issue
[ x ] I’ve verified that I’m running youtube-dl version 2021.12.17
[ x ] I’ve checked that all provided URLs are alive and playable in a browser
[ x ] I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
[ x ] I’ve searched the bugtracker for similar bug reports including closed ones
[ x ] I’ve read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'--verbose', u'--write-info-json', u'https://learning.oreilly.com/videos/linux-shell-scripting/9781789800906/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.16 (CPython) - Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.12
[debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0
[debug] Proxy map: {}
[safari:course] Downloading login page
ERROR: An extractor error has occurred. (caused by KeyError(u'next',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 533, in extract
    self.initialize()
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 437, in initialize
    self._real_initialize()
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/safari.py", line 29, in _real_initialize
    self._login()
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/safari.py", line 51, in _login
    'https://api.oreilly.com', qs['next'][0])
KeyError: u'next'
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 547, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'next',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Issue seems to be with some changes on login page of oreilly site. youtube-dl is not able to login and hence can not proceed further.

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 17 (6 by maintainers)

Commits related to this issue

Check for oreilly login with new url Resolves #30884 — committed to epsilonSpider/youtube-dl by epsilonSpider a year ago

Most upvoted comments

I managed to download a single video (length bigger than a minute).

Edit: I meant that I manage to download a whole course where at least one of the videos is bigger than a minute.

hunter86bg on Feb 12, 2023

Use --cookies ... with a cookie file from your logged-in browser session, and don’t use -u/--username ....

In order to archive more than just 2 minutes of video; there was a temporary fix found that when using both cookies & username/password, it would allow full-length archiving of the video.

prof-ninjason on May 5, 2022

Use --cookies ... with a cookie file from your logged-in browser session, and don’t use -u/--username ....

dirkf on Apr 22, 2022

From my understanding, the problem stands in (at least) two places:

If you don´t provide username/password through the options, the user is considered not logged in even if credentials are passed using cookies (https://github.com/ytdl-org/youtube-dl/blob/f24bc9272e9b74efc4c4af87c862f5f78921d424/youtube_dl/extractor/safari.py#L33)

        username, password = self._get_login_info()
        if username is None:
            return

Apparently the is_logged function is not working properly too. This PR should fix this issue https://github.com/ytdl-org/youtube-dl/pull/31524

I tried locally to replace the is_logged function

        def is_logged(urlh):
            url = urlh.geturl()
            parsed_url = compat_urlparse.urlparse(url)
            return parsed_url.hostname.endswith('learning.oreilly.com') and (
                parsed_url.path.startswith('/home/')
                or (parsed_url.path == '/member/login/' and not parsed_url.query))

And this in combination with the -u someuser -p somepass --cookies <cookies-file> seems to solve the problem

Kerruba on Jul 17, 2023

Confirmed.

I refreshed my cookies. I removed username/password. I have evaluated my script into a single one-line command just to make sure it wasn’t my own personal script. I also removed the last / in the URL just in case.

Still having a “next” error.

prof-ninjason on May 3, 2022

Still have an error: [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u’–cookies’, u’./cookies.txt’, u’–verbose’, u’–write-info-json’, u’https://learning.oreilly.com/videos/complete-bash-shell/9781800209695/'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2021.12.17 [debug] Python version 2.7.16 (CPython) - Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.12 [debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0 [debug] Proxy map: {} [safari:course] 9781800209695: Downloading course JSON [download] Downloading playlist: Complete Bash Shell Scripting [safari:course] playlist Complete Bash Shell Scripting: Collected 93 video ids (downloading 93 of them) [download] Downloading video 1 of 93 [safari:api] 9781800209695/video1_1: Downloading part JSON [safari] 9781800209695-video1_1: Downloading webpage ERROR: Unable to extract kaltura reference id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File “/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py”, line 815, in wrapper return func(self, *args, **kwargs) File “/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py”, line 836, in __extract_info ie_result = ie.extract(url) File “/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py”, line 534, in extract ie_result = self._real_extract(url) File “/usr/local/bin/youtube-dl/youtube_dl/extractor/safari.py”, line 147, in _real_extract webpage, ‘kaltura reference id’, group=‘id’) File “/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py”, line 1012, in _search_regex raise RegexNotFoundError(‘Unable to extract %s’ % _name) RegexNotFoundError: Unable to extract kaltura reference id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Btw, it was working fine with Oreilly a few days back.

sandygmaharaj on Apr 25, 2022