youtube-dl: [BUG] Missing video(s) in channel download on YouTube
Please follow the guide below
- You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
- Put an
xinto all the boxes [ ] relevant to your issue (like this:[x]) - Use the Preview tab to see what your issue will actually look like
Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.04.16. If it’s not, read this FAQ entry and update. Issues with outdated version will be rejected.
- I’ve verified and I assure that I’m running youtube-dl 2018.04.16
Before submitting an issue make sure you have:
- At least skimmed through the README, most notably the FAQ and BUGS sections
- Searched the bugtracker for similar issues including closed ones
- Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
What is the purpose of your issue?
- Bug report (encountered problems with youtube-dl)
- Site support request (request for adding support for a new site)
- Feature request (request for a new functionality)
- Question
- Other
This YouTube channel (https://www.youtube.com/user/ChildrenofPoseidon2) has 3 videos, but youtube-dl only finds 2 videos.
Command: youtube-dl -s -v https://www.youtube.com/user/ChildrenofPoseidon2
Log output:
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.youtube.com/user/ChildrenofPoseidon2', u'-s', u'-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.04.16
[debug] Python version 2.7.10 (CPython) - Darwin-17.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.2, ffprobe 3.4.2
[debug] Proxy map: {}
[youtube:user] ChildrenofPoseidon2: Downloading channel page
[youtube:playlist] UUyQUl9M6OfRX7EKhi8SPN_A: Downloading webpage
[download] Downloading playlist: Uploads from Jeffabel and Friends Extras
[youtube:playlist] playlist Uploads from Jeffabel and Friends Extras: Downloading 2 videos
[download] Downloading video 1 of 2
[youtube] 55DF8tRSMco: Downloading webpage
[youtube] 55DF8tRSMco: Downloading video info webpage
[youtube] 55DF8tRSMco: Extracting video information
[debug] Default format spec: bestvideo+bestaudio/best
[download] Downloading video 2 of 2
[youtube] Lm-_ZUiEQDk: Downloading webpage
[youtube] Lm-_ZUiEQDk: Downloading video info webpage
[youtube] Lm-_ZUiEQDk: Extracting video information
[debug] Default format spec: bestvideo+bestaudio/best
[download] Finished downloading playlist: Uploads from Jeffabel and Friends Extras
This video (https://www.youtube.com/watch?v=tZPI1mmVPCk) is missing from the output.
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 1
- Comments: 18
The rationale behind delegation to playlist extractor is clearly explained in the comment above the code mentioned: user rendition has a limitation of 100 pages with 30 videos per page (35 pages at the moment of writing the code), at the same time playlist has no such limitation and provides unlimited 100 videos per page rendition (see https://www.youtube.com/user/drighk/videos). “Fixing” it by removing the delegation will make it impossible to download whole channels with >3k videos.
Someone recently managed to reproduce this on Reddit here: https://www.reddit.com/r/DataHoarder/comments/9qrlbp/i_wrote_a_pythonselenium_based_crawler_to_really/?st=jnmcv30j&sh=66312301
They were trying to download this user:
Pasting my analysis here:
Just trying some things to see what’s up here:
This yields:
Which means youtube-dl converted that user to this playlist:
Which uses the exact same ID as the channel in all of yotube’s links (except a different prefix)
The channel’s id is this:
Which gets converted with this code:
From this line in YoutubeChannelIE: https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/youtube.py#L2485
YoutubeChannelIE is inherited by YoutubeUserIE, (probably because there is so much in common between the two). I would gather that their assumption – that this URL is the most ideal place to find all the channel’s videos – is probably an incorrect assumption in youtube-dl.
Imo this looks like a fairly drastic bug, but it shouldn’t be too hard to fix.
Edit: Ironically even Youtube’s “Play All” button on the user’s “Uploads” page jumps to a playlist with only 22 videos too, despite there being more than 60 uploads in that playlist. This might be somewhat exasperated by a bug on Youtube’s server:
https://www.youtube.com/user/LanaDelRey/videos
As I understand it, the problem is that live videos are not added to the channel playlist. I have a suggestion - add some parameter to force the video to the specified address (with the parameters specified by the user). For example, if you specify the URL https://www.youtube.com/channel/UCjzHeG1KWoonmf9d5KBvSiw/videos?view=2, then only live broadcasts will be output.
There is also a fairly elegant way to solve a problem without the need to add new parameters. But it is less universal.
It makes no sense to load a playlist if the --playlist-end <n> parameter is specified, where n is less than the number of videos on one page. In addition, it will reduce the number of requests to the server YouTube.
I think this is quite a serious problem, since live streams on YouTube are now used very often and do not always persist after they end. The only way to record them is to record it during the live broadcast.