yt-dlp: Do not skip fragments on filesystem errors
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I intentionally remove or skip any mandatory* field
Checklist
- I’m reporting a bug unrelated to a specific site
- I’ve verified that I’m running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
Provide a description that is worded well enough to be understood
When disk space becomes insufficient, yt-dlp says:
ERROR: unable to write data: [Errno 28] No space left on device
OSError: [Errno 28] No space left on device
[download] fragment not found; Skipping fragment 330 ...
In most cases, it doesn’t make sense to skip the fragment, because:
- the following fragments will suffer from the problem of insufficient disk space and
- a part of the download will be missing and/or
- the audio/video may become out-of-sync if the fragment was from an audio-only or video-only track.
solution
When running out of disk space, yt-dlp should either
- pause to give the user a chance to free some disk space or
- abort, so that the user can resume the download after having freed some disk space
related issues
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>) - If using API, add
'verbose': TruetoYoutubeDLparams instead - Copy the WHOLE output (starting with
[debug] Command-line config) and insert it below
[debug] Command-line config: ['https://dms.redbull.tv/v5/destination/stv/AAWCVHH74J0J0A0TBSDD/personal_computer/http/at/de_AT/playlist.m3u8', '-f-']
[debug] Portable config "c:\hp\-\!portableapps\yt-dlp\yt-dlp.conf": ['--verbose', '--format-sort', 'hasvid,ie_pref,quality,res,hdr,codec,size,br,asr,ext,fps,hasaud,lang,source,proto,id', '-f', 'bestvideo*+bestaudio/best', '-f-', '--paths', 'c:\\hp\\-\\#YT', '--ffmpeg-location', '.\\helpertools\\FFmpeg\\bin', '-o', '.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'thumbnail:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\thumbs\\thumb.%(ext)s', '-o', 'subtitle:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\subs\\%(alt_title,title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'infojson:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(alt_title,title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'link:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'description:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'annotation:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'pl_infojson:.\\%(playlist|)s\\playlist.%(ext)s', '-o', 'pl_thumbnail:.\\%(playlist|)s\\folder.%(ext)s', '-o', 'pl_description:.\\%(playlist|)s\\playlist description.%(ext)s', '--output-na-placeholder', '~~~', '--write-info-json', '--write-description', '--write-annotations', '--no-overwrites', '--no-post-overwrites', '--merge-output-format', 'mkv', '--remux-video', 'mkv', '--write-subs', '--write-auto-subs', '--all-subs', '--sub-langs', 'all', '--sub-format', 'all/ttml/vtt/best', '--write-thumbnail', '--write-all-thumbnails', '--embed-metadata', '--embed-chapters', '--no-embed-info-json', '--write-link', '--write-url-link', '--write-desktop-link', '--no-check-certificate', '--retries', 'infinite', '--geo-bypass', '--geo-bypass-country', 'AT', '--no-clean-infojson', '--write-comments', '--audio-multistreams', '--no-playlist']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k 25 Mar 2021)
[debug] exe versions: ffmpeg N-110185-gb564ad8eac-20230407 (setts), ffprobe N-110185-gb564ad8eac-20230407
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Using fake IP 77.119.119.20 (AT) as X-Forwarded-For
[generic] Extracting URL: https://dms.redbull.tv/v5/destination/stv/AAWCVHH74J0J0A0TBSDD/personal_computer/http/at/de_AT/playlist.m3u8
[generic] playlist: Downloading webpage
[debug] Identified a direct video link
[generic] playlist: Downloading m3u8 information
[debug] Sort order given by user: hasvid, ie_pref, quality, res, hdr, codec, size, br, asr, ext, fps, hasaud, lang, source, proto, id
[debug] Formats sorted by: hasvid, ie_pref, quality, res, hdr, vcodec, acodec, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, fps, hasaud, lang, source, proto, id, channels
[info] Available formats for playlist:
ID EXT RESOLUTION FPS │ TBR PROTO │ VCODEC VBR ACODEC MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────
audio-stereo-aac-Deutsch mp4 audio only │ m3u8 │ audio only unknown [de] Deutsch
518 mp4 426x240 25 │ 518k m3u8 │ avc1.4D401E 518k video only
848 mp4 640x360 25 │ 848k m3u8 │ avc1.4D401E 848k video only
1507 mp4 960x540 25 │ 1507k m3u8 │ avc1.4D401F 1507k video only
2927 mp4 1280x720 25 │ 2928k m3u8 │ avc1.4D401F 2928k video only
6152 mp4 1920x1080 25 │ 6152k m3u8 │ avc1.640028 6152k video only
Enter format selector: bv+ba
[info] playlist: Downloading 1 format(s): 6152+audio-stereo-aac-Deutsch
[info] There's no video description to write
[info] There's no subtitles for the requested languages
[info] There's no video thumbnails to download
[info] Writing video metadata as JSON to: c:\hp\-\#YT\#nodate# playlist\!nfo\playlist [generic.Generic.playlist] [6152+audio-stereo-aac-Deutsch].info.json
WARNING: There are no annotations to write.
[info] Writing internet shortcut (.url) to: c:\hp\-\#YT\#nodate# playlist\!nfo\playlist [generic.Generic.playlist] [6152+audio-stereo-aac-Deutsch].url
[info] Writing internet shortcut (.desktop) to: c:\hp\-\#YT\#nodate# playlist\!nfo\playlist [generic.Generic.playlist] [6152+audio-stereo-aac-Deutsch].desktop
[debug] Invoking hlsnative downloader on "https://dms.redbull.tv/v5/dms/media/vod/stv/AAWCVHH74J0J0A0TBSDD/1920x1080@7292803/personal_computer/http/at/playlist.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 620
[download] Destination: c:\hp\-\#YT\#nodate# playlist\playlist [generic.Generic.playlist] [6152+audio-stereo-aac-Deutsch].f6152.mp4
[debug] File locking is not supported. Proceeding without locking
[download] 53.1% of ~ 1.29GiB at 75.04KiB/s ETA 01:36:08 (frag 329/620)
ERROR: unable to write data: [Errno 28] No space left on device
Traceback (most recent call last):
File "yt_dlp\downloader\http.py", line 288, in download
OSError: [Errno 28] No space left on device
[download] fragment not found; Skipping fragment 330 ...
[download] 54.0% of ~ 1.29GiB at 138.58KiB/s ETA 01:34:39 (frag 334/620)
ERROR: Interrupted by user
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 30 (14 by maintainers)
While I have no idea what chrizilla really wants, I did find something related that I would like to see…
I’d like missing fragments to be tried to obtained even after all other have been downloaded. Although…
The current way files are built from fragments would need to be completely re-written in order for it to be possible.
If you are wondering why it would be helpful over just aborting and trying another time: It would help downloading from unreliable sources where some fragments are missing at certain times, similar to incompletely seeded torrents. It would allow you to get all the ones that are available so next time you try you don’t have to worry about the remaining fragments. Although I have to admit, it’s the the most niche use-case ever…
Let me please mull this over and come back to you hopefully tomorrow.
Hidden
Let others worry about their own use-cases - https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-anyone-going-to-need-the-feature
The option description exist so you don’t have to try and infer meaning from the switch names alone! The aliases are so that all boolean options have a
--no-variant.If a fragment is unavailable after retries are exhausted, we can either skip it, or abort download. I don’t understand what other choice is there. Can you explain how you expect “don’t abort and don’t skip” to work? It makes no sense to me.
It was made during yt-dlp’s inception and so I didn’t have to worry about users already relying on the behavior. Due to the current popularity of yt-dlp, new changes to default will be only made after very careful consideration, when there is no other reasonable option, with warning (and preferably a deprecation period), and with compat options to revert the change. There are so many things I’d change if this were a new project, but breaking user-space is a serious matter.
That said, I wouldn’t consider what you originally asked for to be a breaking change - hence why I marked this as a valid feature request. But now it seems to me like you are talking about totally different request - could you explain the link that I could be missing?
yt-dlp does write a
.ytdlfile, which is how it knows where to continue from. In that file, it’d have written that the skipped fragments have already been tried. So it won’t try to download them again. For network errors, this is correct behavior.This is already possible with
--retries inf --fragment-retries inf --retry-sleep http,fragment:expBut your original request is contradicting this…
I was thinking we could force
--abort-on-unavailable-fragmentand use--file-access-retriesinstead forOSErrors like “disk full”. But now I am utterly confused about your requirement. Pls explain what you want clearly before proposing solutions.In my OP I suggested pause or abort as possible solutions but I had other users in mind when I wrote that. Not myself.
Can I please lay out what I want yt-dlp to do (personally in my case) and maybe based on that there will be a useful upshot for other users as well. So what I am trying to achieve is this:
In my case it is simple, because I want yt-dlp to behave the same in both situations (write error AND connection error): Try infinitely with exponential wait time.
(One possible exception probably are livestreams, but let's put that aside for a moment, because the issue is already complicated enough. But I think different rules need to apply there, because for obvious reasons in a livestream you cannot endlessly for a fragment to become available or writable.)
So now I am trying to use:
What’s a bit unclear is how to use
--abort-on-unavailable-fragments. The helpfile says:This is very confusing, because what are presented here as aliases, actually convey two opposing instructions:
--abort-on-unavailable-fragments: Abort download if a fragment is unavailable.--no-skip-unavailable-fragments: Don’t abort but continue downloading without skipping unavailable fragments.This is confusing, because two very different decisions are mixed:
Taking this into account, the currently contradicting parameters should be changed to:
--abort-on-unavailable-fragments: Abort download if a fragment is unavailable.--no-abort-on-unavailable-fragments: Doesn’t abort (but continues) download if a fragment is unavailable.--skip-unavailable-fragments: Skip unavailable fragments.--no-skip-unavailable-fragments: Doesn’t skip unavailable fragments (but continues to try as long as defined by--retry-sleep exp=START[:END])So in my case I would need both:
But I cannot use these parameters concurrently as they are currently erroneously defined as opposites of each other, when they are clearly not, as is obvious from my case.
So I would strongly urge to redefine the abovementioned parameters. In doing so, several different user wishes can be handled by yt-dlp when encountering unavailable fragments:
--abort-on-unavailable-fragments--no-abort-on-unavailable-fragments --skip-unavailable-fragments--no-abort-on-unavailable-fragments --no-skip-unavailable-fragmentsThis should cover all possible user wishes regarding yt-dlp when encountering unavailable fragments, because it covers all 3 possible combinations of abort(or-not) + skip(or-not). (The 4th theoretical combination doesn’t exist, because aborting does away with the question of skipping fragments or not.)
Yes, unfortunately this changes the way yt-dlp used to work, but sometimes this is necessary to repair things and sometimes you already had to choose that path (for example in relation to multistreams which also changed how some parameters used to work).