yt-dlp: Nebula broken
Checklist
- I’m reporting a broken site
- I’ve verified that I’m running yt-dlp version 2022.05.18 (update instructions) or later (specify commit)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
- I’ve read about sharing account credentials and I’m willing to share it if required
Region
United States
Description
It seems that Nebula support is broken. This seems to be a problem with all videos. I know it worked last week because I verified that the change in Issue 3609 worked for me. I’m not on a particular release but did make sure I did a pull from master before I submitted the report.
I’ve verified that I can watch the videos directly via the website with no problems.
Verbose log
stinerman@pc-debian:~/Videos$ ~/bin/yt-dlp -vU https://nebula.app/videos/wendover-how-to-design-a-theme-park-to-take-tons-of-your-money
[debug] Command-line config: ['-vU', 'https://nebula.app/videos/wendover-how-to-design-a-theme-park-to-take-tons-of-your-money']
[debug] User config "/home/stinerman/.config/yt-dlp/config": ['--mark-watched', '--netrc', '--prefer-insecure', '--no-check-certificate']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.05.18 [b14d52355] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: d9473db78
[debug] Python version 3.9.2 (CPython 64bit) - Linux-5.10.0-14-amd64-x86_64-with-glibc2.31
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.3.4-0, ffprobe 4.3.4-0
[debug] Optional libraries: Cryptodome-3.9.7, certifi-2020.06.20, secretstorage-3.3.1, sqlite3-2.6.0
[debug] Proxy map: {}
Latest version: 2022.05.18, Current version: 2022.05.18
yt-dlp is up to date (2022.05.18)
[Nebula] Logging in to Nebula with supplied credentials
[Nebula] Authorizing to Nebula
[Nebula] Retrieving Zype access token
[debug] [Nebula] Extracting URL: https://nebula.app/videos/wendover-how-to-design-a-theme-park-to-take-tons-of-your-money
[Nebula] wendover-how-to-design-a-theme-park-to-take-tons-of-your-money: Fetching video meta data
ERROR: wendover-how-to-design-a-theme-park-to-take-tons-of-your-money: An extractor error has occurred. (caused by KeyError('zype_id')); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U
File "/home/stinerman/src/yt-dlp/yt_dlp/extractor/common.py", line 642, in extract
ie_result = self._real_extract(url)
File "/home/stinerman/src/yt-dlp/yt_dlp/extractor/nebula.py", line 247, in _real_extract
return self._build_video_info(video)
File "/home/stinerman/src/yt-dlp/yt_dlp/extractor/nebula.py", line 119, in _build_video_info
zype_id = episode['zype_id']
KeyError: 'zype_id'
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (15 by maintainers)
@stinerman Thank you for tagging me, I’ll take a look asap!
Oh that is weird!
But you know what, what tipped us off to this change initially was that the
zype_iddisappeared from one of the API calls. That’s what OP originally reported, aKeyError: 'zype_id'.Then when I started working on using the new endpoints, we discussed here that with the Zype ID no longer visible and the videos instead having Nebula-internal IDs, the archive file support would get broken. We agreed that that is problematic. And then when I looked at it again a couple days later, I found a Zype ID field. At the time, I thought I had simply missed it, but that is in fact the very same field that broke the extractor in the first place!
You’re right, they must have brought it back already around May 29, when I wrote:
I think the changes here are ultimately still worth it. But good to know that they aren’t urgent anymore.
@stinerman I can reproduce this. Even the unit tests all fail now, so it’s not just a mistake I made, but rather Nebula changed something in their architecture.
I did a quick analysis. The good news is that the channel support seems fully functional (it just can’t extract any videos, but for the same reason direct video downloads do not work anymore), and the metadata extraction appears completely intact, too.
But it seems that Nebula no longer directly exposes the Zype video ID, which the Nebula extractor currently uses to hand video downloading off to the Zype extractor. Instead, there’s now a
/video/{video_id}/stream/endpoint, which returns a manifest URL. The manifest URL returns an M3U file. So probably, this is the point where we can drop the Zype extractor dependency completely and instead directly hand off the manifest for downloading.It all looks very clean and well-structured on the Nebula API, so I should be able to do this in the next days. But yeah, it does require a code change to restore Nebula support.
tl;dr: Nebula video extractor currently fully broken. Fix is possible, PR will come in the next days.
The PR can still be merged (though with low priority), but closing the issue since it fixed itself
I just wanted to mention that Nebula works again with yt-dlp 2022.05.18 (unpatched), they must have rolled back that change?
@Lamieur The delay wasn’t caused by the pondering, but rather because this breakage ‘unfortunately’ happened to coincide with a previously planned vacation on my end. 😉
But I totally get what you’re saying. I’m back since today and once the kids return to kindergarden on Tuesday, I’m hoping to be able to wrap this up quickly!
While you ponder this, can we just get a fix for watching a single video? I’m all for fixing needless queries for channels/playlists, but that’s not a show-stopper for fixing a “nothing works” condition 😉
(I know, I know - if you don’t fix this now, it will stay like that forever. We’ve all been there 😕)
FYI on this @hheimbuerger. This is a friendly ping because I know you do a lot of the maintenance. Thank you for that.