youtube-dl: «File name too long» should not error

Checklist

  • I’m reporting a feature request
  • I’ve verified that I’m running youtube-dl version 2021.06.06
  • I’ve searched the bugtracker for similar feature requests including closed ones

Description

To reproduce:

  • try to download any video with a really long title
  • you get ERROR: unable to open for writing: [Errno 36] File name too long:
  • the video is not downloaded

My suggestion:

  • instead of an ERROR put a WARNING
  • download the video regardless, just truncate the title to an appropriate size, similar to what suggested in #29627

This will improve UX, check bug tracker to see how common this issue is.

#7765 talks about this too (with a linked pull request) but:

  • does not elaborate on whether it is actually a good thing / possible drawbacks
  • is limited to collaborators

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 7
  • Comments: 33 (17 by maintainers)

Commits related to this issue

Most upvoted comments

But my ticket is addressing a different point. Shrinking/truncating long filenames should be default. It would be an immense step forward in terms of UX to go to «it just works, emits a warning» from «it does not work, errors out; search the error / skim the FAQ».

Of course, if the user has specific needs, they should be able to turn this off and get an error. (maybe via --fixup option).

Il 06 settembre 2021 alle 19:33 dirkf ha scritto:

The general solution to this is a FAQ entry

### My download fails because youtube-dl is trying to write to a file whose name is too long for the file system.

Use the `--get-filename` option to see what filename is being proposed. Then adjust, or add, the `-o`/`--output-template` option in your original youtube-dl command to make the filename shorter. For instance, if the extractor for the site sets a very long `title` field and you are using the default output template `%(title)s-%(id)s.%(ext)s`. you could try `-o %(title).50s-%(id)s.%(ext)s` instead to limit length of the filename.

– You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/ytdl-org/youtube-dl/issues/29912#issuecomment-913945852

This is particularly annoying when downloading videos from Twitter as it takes the whole post as file name. I usually use --id and rename the file manually.

Yeah, having the tool just fail on long file names without the user applying different parameters to mitigate the issue is very rational and an awesome user experience. I mean just defaulting to a template that causes video downloads not to fail with reasonable truncation would be ridiculous. The legions of people coming to the Issues and hopefully reading the FAQ before creating this issue repeatedly shows the wisdom of this decision. 👍

Plainly it’s not a showstopping issue but it is one that generates repeated complaints. It’s still open in case someone is eventually intrigued enough to work out a solution that meets an adequate subset of the problem constraints, after spending however long it takes locked in the basement with the necessary cold towels wrapped round the head. But we should fix Twitter/X if the extractor is ever viable and updated post-Musk.

--trim-filenames is a poorly implemented solution from youtube-dlc and I would not recommend porting it as-is. Even in yt-dlp, I recommend users to use the % formatting to control filename length instead. E.g. https://github.com/yt-dlp/yt-dlp/issues/3494

In the case of Twitter, id is NOT being truncated, which is a plus…

This is actually a bug and shows how poorly implemented the option is! See https://github.com/yt-dlp/yt-dlp/issues/2314. Do not rely on this behavior…

The general solution to this is a (new) FAQ entry

### My download fails because youtube-dl is trying to write to a file whose name is too long for the file system.

Use the `--get-filename` option to see what filename is being proposed. Then adjust, or add, the `-o`/`--output` option in your original youtube-dl command to make the filename shorter. For instance, if the extractor for the site sets a very long `title` field and you are using the default output template `%(title)s-%(id)s.%(ext)s`. you could try `-o %(title).50s-%(id)s.%(ext)s` instead to limit the length of the filename.

I don’t see that --trim-filenames ... is much of an improvement on setting an explicit length for a problem field (title, I’m looking at you) in the output template.

A good solution would:

  • never write a file whose pathname is too long for the filesystem (even for intermediate files)
  • apply generally regardless of extractor and output template combinations
  • not require user intervention, such as having to guess a parameter value and pass it to avoid filename length errors
  • not alter any ID fields in the output template
  • any more?

Clearly these are not completely consistent requirements, as an output template might be, say, %(id)16777216s.%(ext)s. Some prioritisation is necessary.

I still hold that the excessive filename length provoked by Twitter is an extractor bug.

It works the other way. Just set maximum lengths for the fields in the output template that will meet the limits of your OS/filesystem, as well as any others to which you might copy the downloaded file. You should do this for the documented default (set in a config file), and for any specific output template that you pass in --output/-o .... Then any mutilation of template fields is entirely under your control.

The combined abilities of the many contributors to this project and yt-dlp have not so far found an acceptable solution to the general problem posed by this issue. I don’t expect the issue to be closed soon or ever, but who knows?

Some fields are more important than others, like the unique ID’s, hashes, and file extensions can never be allowed to be truncated. As few as possible known fields could have an importance priority floating point number internally assigned in the extractor.

^ Then, generally, you can truncate all other fields equally to fit, as it’ll be more important to have some knowledge of the contents in the filename than have none at all, and 125, not to mention 255, chars per segment (or even a full path) is a lotta text. If the User is hellbent on changing a sensible default into something very much non-sensible like ID + hash + keywords/tags + collection name + description, that’s their fault.

Something like that.

@bryanpaget, you might like to reveal more details of how you “[ran] into this issue”, or is it just Twitter again?

Also, feel free to review the discussion in PR #29989 to understand why this is a difficult issue to fix retrospectively (other than by following the approach in https://github.com/ytdl-org/youtube-dl/issues/29912#issuecomment-913945852), and possibly contribute to the design of a general solution.

The defaults really do work in the vast majority of cases.

You can write a config file with your preferred options and select it with --config-location in your script if some default setting should not be satisfactory (or just make it your default config).

For instance, you could put -o %(title).50s-%(id)s.%(ext)s to avoid downloads being broken by unreasonably long titles.

If anyone has a suggestion for what (reliably concise) metadata item could be used as the title for media extracted from Twitter, we could make a PR for Twitter specifically.

I don’t see any other reasonable solution for the Twitter extractor. Maybe it could pre-truncate the post? That’s not any better than just doing it later when constructing the output file name though.

I’ve run into this issue many times, as I usually want to use youtube-dl to quickly download a video to my downloads folder and I don’t really care about the name but would prefer it give as much information as possible (so I don’t want to just default to using the id).

I would really, really love it if somebody made a youtube-dl flag --just-download-it-seriously-i-dont-care-about-the-details-just-do-your-best-i-believe-in-you, bonus points if that’s the default, but even just having it available so I can write a wrapper script or bash alias that uses it would be amazing.

#30839: “if you were actually running yt-dlp …”

Also, 90 isn’t a “constant”: pick a number that works.

What is this? The 90s? This shouldn’t be an error at all.

Depends on who/what you blame.

I’ve just tonight run into this bug with the Twitter extractor. You can reproduce it with the following:

youtube-dl https://twitter.com/TulsiGabbard/status/1555878318469091330
[twitter] 1555878318469091330: Downloading guest token
[twitter] 1555878318469091330: Downloading JSON metadata
[twitter] 1555878318469091330: Downloading m3u8 information
ERROR: unable to open for writing: [Errno 36] File name too long: 'Tulsi Gabbard 🌺 - Puberty-blocking procedures promoted by the Biden_Harris Admin are child abuse. The FDA has recently confirmed these hormones_drugs have extremely dangerous side effects, like brain swelling and vision loss.-1555878318469091330.mp4.part'

Would you be open to a Twitter specific workaround patch? Perhaps the title could be changed to a combination of the uploader name and date, with the tweet text moved to the description field?

The previous discussions on this point show that the maintainers (Sergey, at least) won’t accept any solution that they see as ad hoc. The project sees the id field in particular as inviolable, and won’t accept a solution that might accidentally truncate an id embedded in a filename. However, if the project’s maintainers should be permanently absent, the project might be adopted by maintainers with different views.

It could have been reasonable to add some sort of filename truncation functionality to the --restrict-filenames option, where it has already been accepted to modify embedded fields, but the existing definition of that option doesn’t allow it.

The cross-platform Python APIs don’t offer a way of telling if the proposed file name is too long. In general this is a tricky issue because the error is only found once the OS kernel has called into the filesystem driver for the device that is to be written to.

We could try modifying sanitize_open() in utils.py, s/t like below:

def sanitize_open(filename, open_mode):
    """Try to open the given filename, and slightly tweak it if this fails.

    Attempts to open the given filename. If this fails, it tries to change
    the filename slightly, step by step, until it's either able to open it
    or it fails and raises a final exception, like the standard open()
    function.

    It returns the tuple (stream, definitive_file_name).
    """
    try:
        if filename == '-':
            if sys.platform == 'win32':
                import msvcrt
                msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
            return (sys.stdout.buffer if hasattr(sys.stdout, 'buffer') else sys.stdout, filename)
        stream = open(encodeFilename(filename), open_mode)
        return (stream, filename)
    except (IOError, OSError) as err:
        if err.errno in (errno.EACCES,):
            raise
        # --- addition ---
        # would go in `compat.py`
        try:
            from textwrap import shorten as compat_textwrap_shorten
        except ImportError:
            def compat_textwrap_shorten(
                    text, width, fix_sentence_endings=False, break_long_words=True,
                    break_on_hyphens=True, placeholder=' [...]'):
                import textwrap
                try_text = textwrap.wrap(
                    text, width,
                    fix_sentence_endings=fix_sentence_endings,
                    break_long_words=break_long_words,
                    break_on_hyphens=break_on_hyphens)
                if len(try_text) == 1:
                    return try_text[0]
                return textwrap.wrap(
                    text, width - len(placeholder),
                    fix_sentence_endings=fix_sentence_endings,
                    break_long_words=break_long_words,
                    break_on_hyphens=break_on_hyphens)[0]+placeholder

        # function would go in utils.py
        def reduce_filename(fname):
            ellipsis = '[...]'
            fname = os.path.split(fname)
            fname = fname[:1] + os.path.splitext(fname[1])
            fname[1] = remove_end(fname[1], ellipsis)
            flen = len(fname[1])
            if flen < 20:
                # give up
                return None
            # might try reduction by less than 2?
            fname[1] = compat_textwrap_shorten(fname[1], 1+flen/2, placeholder=ellipsis)
            return os.path.join(fname[0], ''.join(fname[1:]))

        alt_filename = filename
        while err.errno in (errno.ENAMETOOLONG,):
            alt_filename = reduce_filename(alt_filename)
            if not alt_filename:
                break
            try:
                stream = open(encodeFilename(alt_filename), open_mode)
                return (stream, alt_filename)
            except (IOError, OSError) as err:
                pass
        else:
            # not too long, but still wrong
            filename = alt_filename
        # --- end addition ---
        
        # In case of error, try to remove win32 forbidden chars
        alt_filename = sanitize_path(filename)
        if alt_filename == filename:
            raise
        else:
            # An exception here should be caught in the caller
            stream = open(encodeFilename(alt_filename), open_mode)
            return (stream, alt_filename)

Concerns here:

  • which of ‘invalid filename’ or ‘filename too long’ gets priority if both apply?
  • might other error conditions be signalled (eg, ENOENT, EINVAL) instead of ENAMETOOLONG?

There might also be additional guidance for developers to avoid extracting title and id fields that might exceed typical filename lengths.