yt-dlp: Intermittent "Incomplete data received" errors when downloading YouTube playlist JSON
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I intentionally remove or skip any mandatory* field
Checklist
- I’m reporting that yt-dlp is broken on a supported site
- I’ve verified that I’m running yt-dlp version 2023.09.24 (update instructions) or later (specify commit)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
- I’ve read about sharing account credentials and I’m willing to share it if required
Region
United Kingdom
Provide a description that is worded well enough to be understood
When running the following command:
yt-dlp -vU --dump-single-json --flat-playlist PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf
it sometimes succeeds, and sometimes fails with yt_dlp.utils.ExtractorError: Incomplete data received
as can be seen below. It does not always fail; this error seems to be intermittent. It is also not specific to this playlist; I have seen it fail with the same error on other playlists as well. (The specific playlist mentioned above is https://www.youtube.com/playlist?list=PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf for convenience of linking; it opens in a browser successfully.) In running this command 5 times, it succeeded twice and failed three times.
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>
) - If using API, add
'verbose': True
toYoutubeDL
params instead - Copy the WHOLE output (starting with
[debug] Command-line config
) and insert it below
Complete Verbose Output
[debug] Command-line config: ['-vU', '--dump-single-json', '--flat-playlist', 'PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.09.24 [088add956] (zip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-84-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, pyxattr-0.7.2, secretstorage-3.3.1, sqlite3-3.37.2
[debug] Proxy map: {}
[debug] Loaded 1886 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.09.24, Current version: stable@2023.09.24
Current Build Hash: 9c32c2f215a58c33369ec6105b7e97871905b8d757407bea2bc71d265f9f6cfe
yt-dlp is up to date (stable@2023.09.24)
[youtube:playlist] Extracting URL: PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf
[youtube:tab] Extracting URL: https://www.youtube.com/playlist?list=PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf
[youtube:tab] PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf: Downloading webpage
[youtube:tab] PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf: Redownloading playlist API JSON with unavailable videos
[download] Downloading playlist: Exandria Unlimited: Kymal
[youtube:tab] PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf page 1: Downloading API JSON
WARNING: [youtube:tab] Incomplete data received. Retrying (1/3)...
[youtube:tab] PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf page 1: Downloading API JSON
WARNING: [youtube:tab] Incomplete data received. Retrying (2/3)...
[youtube:tab] PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf page 1: Downloading API JSON
WARNING: [youtube:tab] Incomplete data received. Retrying (3/3)...
[youtube:tab] PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf page 1: Downloading API JSON
ERROR: Incomplete data received
Traceback (most recent call last):
File "/home/aquarius/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 1567, in wrapper
return func(self, *args, **kwargs)
File "/home/aquarius/bin/yt-dlp/yt_dlp/utils/_utils.py", line 2427, in <lambda>
File "/home/aquarius/bin/yt-dlp/yt_dlp/utils/_utils.py", line 2195, in __getitem__
File "/home/aquarius/bin/yt-dlp/yt_dlp/extractor/youtube.py", line 4933, in _entries
response = self._extract_response(
File "/home/aquarius/bin/yt-dlp/yt_dlp/extractor/youtube.py", line 944, in _extract_response
for retry in self.RetryManager():
File "/home/aquarius/bin/yt-dlp/yt_dlp/utils/_utils.py", line 5112, in __iter__
File "/home/aquarius/bin/yt-dlp/yt_dlp/extractor/common.py", line 3760, in _error_or_warning
RetryManager.report_retry(
File "/home/aquarius/bin/yt-dlp/yt_dlp/utils/_utils.py", line 5120, in report_retry
yt_dlp.utils.ExtractorError: Incomplete data received
[youtube:tab] Playlist Exandria Unlimited: Kymal: Downloading 3 items of 2
[debug] The information of all playlist entries will be held in memory
[download] Downloading item 1 of 3
[download] Downloading item 2 of 3
[download] Finished downloading playlist: Exandria Unlimited: Kymal
{"id": "PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf", "title": "Exandria Unlimited: Kymal", "availability": "public", "channel_follower_count": null, "description": "", "tags": [], "thumbnails": [{"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEWCKgBEF5IWvKriqkDCQgBFQAAiEIYAQ==&rs=AOn4CLBT32UGrM2Tqpe7s33qdytjS7P9Zg", "height": 94, "width": 168, "id": "0", "resolution": "168x94"}, {"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEWCMQBEG5IWvKriqkDCQgBFQAAiEIYAQ==&rs=AOn4CLCmcD3sfXb7oMv_RafJN03k7Zx70Q", "height": 110, "width": 196, "id": "1", "resolution": "196x110"}, {"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEXCPYBEIoBSFryq4qpAwkIARUAAIhCGAE=&rs=AOn4CLAbstB9qmaKsecRtnoPYyVadAwn6w", "height": 138, "width": 246, "id": "2", "resolution": "246x138"}, {"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEXCNACELwBSFryq4qpAwkIARUAAIhCGAE=&rs=AOn4CLDKX9VuKXlRu6DyBITHEYunFNvC9w", "height": 188, "width": 336, "id": "3", "resolution": "336x188"}], "modified_date": "20220405", "view_count": 145063, "playlist_count": 2, "channel": "Critical Role", "channel_id": "UCpXBGqwsBkpvcYjsJBQ7LEQ", "uploader_id": "@criticalrole", "uploader": "Critical Role", "channel_url": "https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ", "uploader_url": "https://www.youtube.com/@criticalrole", "_type": "playlist", "entries": [{"_type": "url", "ie_key": "Youtube", "id": "l7SKWVq-jIg", "url": "https://www.youtube.com/watch?v=l7SKWVq-jIg", "title": "Exandria Unlimited: Kymal | Part 1", "description": null, "duration": 15966, "channel_id": "UCpXBGqwsBkpvcYjsJBQ7LEQ", "channel": "Critical Role", "channel_url": "https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ", "uploader": "Critical Role", "uploader_id": "@criticalrole", "uploader_url": "https://www.youtube.com/@criticalrole", "thumbnails": [{"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEbCKgBEF5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLCTHrIg2dsWhdhBUczkJ6YCtIvWuA", "height": 94, "width": 168}, {"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEbCMQBEG5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLAPPsyHJ_zLbINH5Bj_EizUd_JBFA", "height": 110, "width": 196}, {"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEcCPYBEIoBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLBLXJCGJa__hs4kKBdezOgJXLrdgA", "height": 138, "width": 246}, {"url": "https://i.ytimg.com/vi/l7SKWVq-jIg/hqdefault.jpg?sqp=-oaymwEcCNACELwBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLBMxPROxnMFhsLAEHXBwE-ArJrZFg", "height": 188, "width": 336}], "timestamp": null, "release_timestamp": null, "availability": null, "view_count": 903000, "live_status": null, "channel_is_verified": null, "__x_forwarded_for_ip": null}, {"_type": "url", "ie_key": "Youtube", "id": "E-0bSdoPj5o", "url": "https://www.youtube.com/watch?v=E-0bSdoPj5o", "title": "Exandria Unlimited: Kymal | Part 2", "description": null, "duration": 17645, "channel_id": "UCpXBGqwsBkpvcYjsJBQ7LEQ", "channel": "Critical Role", "channel_url": "https://www.youtube.com/channel/UCpXBGqwsBkpvcYjsJBQ7LEQ", "uploader": "Critical Role", "uploader_id": "@criticalrole", "uploader_url": "https://www.youtube.com/@criticalrole", "thumbnails": [{"url": "https://i.ytimg.com/vi/E-0bSdoPj5o/hqdefault.jpg?sqp=-oaymwEbCKgBEF5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLCtABgAEqt2uIMHBNmGOl-QYtewSg", "height": 94, "width": 168}, {"url": "https://i.ytimg.com/vi/E-0bSdoPj5o/hqdefault.jpg?sqp=-oaymwEbCMQBEG5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLA8CX3O2ydFXuzOF7XerNu8nzjgUw", "height": 110, "width": 196}, {"url": "https://i.ytimg.com/vi/E-0bSdoPj5o/hqdefault.jpg?sqp=-oaymwEcCPYBEIoBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCGeEicO2uo1HtD_pW5PnqYsWsKQw", "height": 138, "width": 246}, {"url": "https://i.ytimg.com/vi/E-0bSdoPj5o/hqdefault.jpg?sqp=-oaymwEcCNACELwBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCOwRmzk5veeyUtggdMcxjm2xHrcw", "height": 188, "width": 336}], "timestamp": null, "release_timestamp": null, "availability": null, "view_count": 561000, "live_status": null, "channel_is_verified": null, "__x_forwarded_for_ip": null}, null], "extractor_key": "YoutubeTab", "extractor": "youtube:tab", "webpage_url": "https://www.youtube.com/playlist?list=PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf", "original_url": "PL1tiwbzkOjQwgI-BPd0nujKfVkCag3xFf", "webpage_url_basename": "playlist", "webpage_url_domain": "youtube.com", "requested_entries": [1, 2, 3], "epoch": 1695678694, "__files_to_move": {}, "_version": {"version": "2023.09.24", "current_git_head": null, "release_git_head": "088add9567d39b758737e4299a0e619fd89d2e8f", "repository": "yt-dlp/yt-dlp"}}
About this issue
- Original URL
- State: closed
- Created 9 months ago
- Reactions: 13
- Comments: 24 (6 by maintainers)
Commits related to this issue
- Work around YouTube playlist bug. More details over at yt-dlp: https://github.com/yt-dlp/yt-dlp/issues/8206 — committed to sturmen/yt-speeder-upper by sturmen 9 months ago
Doing some digging, this appears to be a bug on YouTube’s end, but we can take some steps in yt-dlp to handle it better.
In YouTube, if you own the playlist, you get a “recommended videos shelf” at the end of the playlist. It seems like yt-dlp tries to request this page (which is expected given the structure), but doesn’t extract any videos (the traverser doesn’t support the path required)
However, it seems like in a recent YouTube update they are adding this continuation for the recommended videos shelf to any playlist - even if you don’t own it. This seems to be a bug on their end since it never returns with any videos/data in this case. You can see this for yourself in the browser - when you load a 1 page playlist that is not your own, there is a divider at the end and there is a spinner for a moment as it tries loads this shelf that has no videos/data.
What I’m thinking of doing:
Incomplete Data
error a warning by default, with an extractor arg toggle to make it an error.- Add a guard to block extraction of recommended videos shelf (after the page is requested) so we don’t accidently extract it if we were to extend our traversal support in the future.not necessary atmNote that this issue is non-fatal, so everything should still work as expected by default.
If you are embedding yt-dlp, set
ignoreerrors
totrue
.Don’t use
--extractor-retries infinite
for YouTubeThe bug is on YouTube’s end and there have been other similar cases where retries don’t help. Just that it’s not always possible to tell the difference between when they can and not.
I can also reproduce this error. I last tested with two playlists. https://www.youtube.com/playlist?list=PLowKtXNTBypFbtuVMUVXNR0z1mu7dp7eH https://www.youtube.com/playlist?list=PLGWGc5dfbzn_uSMm5jfeaPcpDWo5olbN7
What I have observed is that yt-dlp claims that when the error occurs there is one more video in the playlist than there really is.
Without error, 23 videos in playlist, correct number
With error, supposedly 24 videos in playlist, wrong number
I can intermittently reproduce the errors. Today was the first I’ve noticed them
I’m on 2023.11.16 and I get this problem on every Playlist I try to download. While waiting for YouTube to fix it on their end, is there any workaround in the meantime? Any help is deeply appreciated. Thanks for this great piece of software.
It appears to be fixed. I removed my workaround and ran yt-dlp-2023.10.7 Python module both in an outside of a debugger without issue. Thank you for the fix!
Actually, not quite: It still loops through the “Incomplete data received” warning, wasting time and bandwidth, and with
--extractor-retries infinite
, it even gets stuck in an infinite loop.Can confirm, works with the commit from 3 days ago.
In MPV case, setting
ytdl-raw-options-append=ignore-errors=
inmpv.conf
doesn’t seem to do anything, and yeah yt-dlp do use--ignore-errors
.Test case:
yt-dlp’s command line:
"yt-dlp" --no-warnings -J --flat-playlist --sub-format ass/srt/best --format bestvideo+bestaudio/best --quiet --ignore-no-formats-error --yes-playlist --extractor-args youtube:player_skip=webpage,configs,js;player_client=android,web;lang=vi,en --flat-playlist --ignore-errors -- https://www.youtube.com/playlist?list=PLVM1qRtVEeErISdPLpYEc53XEHU5m34cl
I tried to send all kind of error suppressings but the only success was to modify
ytdl_hook.lua
as in https://github.com/mpv-player/mpv/pull/12511No, I’m running my commands through my local isp when observing the issue.
I’m getting this error too. The JSON key
playlist_count
has the correct value, but theentries
array has an extra element at the end (the element has anull
value).If the error happens when downloading the playlist, yt-dlp says
[youtube:tab] playlist name: Downloading 4 items of 3
(for a playlist with 3 items). But when it comes to download the 4th (nonexistent) item, it just says[download] Finished downloading playlist: playlist name
. The last item it downloaded was3 of 4
.For anyone hitting this error, are you running over a VPN? I noticed that running 2023.09.24 on VPN produces the errors but running same version through local ISP succeeds.
I’ve also been experiencing this error with playlists as of the last day or so on 2023.07.06
Note: I am experiencing this on 2023.07.06 as well; not a regression
Seeing same here on latest release