syncthing: Rename special/unsupported characters in filenames

As mentioned in https://github.com/syncthing/syncthing-android/issues/192 , some filenames are not accepted by windows hosts because they contain ‘special characters’ like colons or bars.

@schuft69 suggested to add a “Rename special characters to ‘_’” option to resolve this issue.

< (less than)
> (greater than)
: (colon)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)

About this issue

  • Original URL
  • State: open
  • Created 12 years ago
  • Reactions: 7
  • Comments: 62 (19 by maintainers)

Commits related to this issue

Most upvoted comments

These are now out of sync items, which explain why they are out of sync, so I think the problem is somewhat solved. You know what you need to do, and I’d rather not have dashes and random stuff added to my filenames by some software I am using.

I think it would be good if Syncthing could tackle the problem. It’s the kind of problem that needs to be tackled when building a sync solution for everyone that is easy to use. Most people would not mind if “:” is converted to “_” and those that do could change the default setting, whereas not syncing a file at all has bigger impact.

Dropbox replace trailing spaces in filenames on any platform, when first detected. If you create a file "test " with trailing space on Mac, then as soon as Dropbox detects it (even if you aren’t syncing with any other platforms), it will rename the file locally to “test” without the trailing space.

I am working on a new kind of sync app for Ronomon, and recently worked on replacing reserved characters with underscores.

These are the characters that would need to be replaced to support almost every platform:

  • " Double Quote
    • Asterisk
  • : Colon
  • < Less Than
  • Greater Than

  • ? Question Mark
  • | Pipe
  • NUL Character (Byte 0)
  • Control Characters (Bytes 1-31, Byte 127)
  • Leading Hyphens (cause problems with many command line tools on Linux and Mac)
  • Trailing Dots
  • Trailing Spaces
  • Parent directory alias (…)
  • Current directory alias (.)
  • Home directory alias (~)

These characters should be replaced with as many underscores.

And then these device names also need to be modified slightly because AUX and AUX.txt are invalid on Windows but AUX_.txt is fine:

  • $IDLE$
  • AUX
  • COM1
  • COM2
  • COM3
  • COM4
  • COM5
  • COM6
  • COM7
  • COM8
  • COM9
  • CONFIG$
  • CON
  • CLOCK$
  • KEYBD$
  • LPT1
  • LPT2
  • LPT3
  • LPT4
  • LPT5
  • LPT6
  • LPT7
  • LPT8
  • LPT9
  • LST
  • NUL
  • PRN
  • SCREEN$
  • $AttrDef
  • $BadClus
  • $Bitmap
  • $Boot
  • $LogFile
  • $MFT
  • $MFTMirr
  • pagefile.sys
  • $Secure
  • $UpCase
  • $Volume
  • $Extend

These reserved device names should be automatically appended with an underscore (e.g. “AUX_” or “aux_.txt”).

Please let me know if something is left out here.

If it would be helpful here, then these are some key ideas which might make the problem manageable and not so hard:

  1. Rename the files on any platform as soon as they are spotted. Renaming them only when they reach a platform where they are invalid only delays the rename and may surprise the user later.
  2. The file should be renamed across the cluster. i.e. There should be no special mapping to preserve invalid characters on platforms which allow them.
  3. When renaming the file locally when the file is first detected by the scanner, first check if another file already exists with the proposed rename. If it does, then add a “(Reserved Character Conflict 1)” label to the end of the filename but before the extension, and then try again or increment the conflict count until the destination is unique.
  4. Take care with hidden files (e.g. “.hidden:file”) to not add the conflict label before the period. In this case there is no extension and the conflict label needs to be added on the right hand side of the period not the left hand side (Dropbox gets this wrong).
  5. Rare case: Make sure that after replacing reserved characters, the filename has not been inadvertently converted into an Apple Double file (which starts with “."), if it was not previously an Apple Double file before the replacement. If it is now an Apple Double file, then convert the ".” prefix to “.-”, i.e. use a dash instead of an underscore.
  6. Very rare case: Make sure that after replacing reserved characters, the filename has not been inadvertently converted into a “.DS_Store” file. If it was, then convert the “_” to “-”, i.e. use a dash instead of an underscore.
  7. Very rare case: On Windows, certain short 8.3 filenames with no corresponding long filename and also containing a tilde, such as “SECURE~1.TXT”, can cause rare conflicts with other files such as “SecureSocketsLayer.txt” and “SecureFTPServer.txt”, depending on the order in which they are synced from different machines, if they are resolved by Windows to the same short 8.3 filename. These conflicts cannot be resolved in the usual manner by appending a conflict label to the filename. Instead, the tilde in these short 8.3 filenames should be automatically replaced with an underscore (e.g. “SECURE_1.TXT”) when first detected in filenames on any platform, so that these short 8.3 filenames can continue to be synced.

This should cover:

exFAT (http://en.wikipedia.org/wiki/ExFAT) VFAT (http://en.wikipedia.org/wiki/File_Allocation_Table#VFAT) NTFS (http://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations) HFS+ (http://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations)

For non-case-preserving filesystems (FAT12, FAT16, FAT32), we may also need to replace “+,.;=[]!@” but I have not tested this yet.

Hope this helps.

Should this thread be closed, with a link to a new issue called “Write a Detailed Specification for How to Reliably Translate and Sync Filenames that are Illegal on Some Platforms”?

We can at least sketch out what that spec has to contain before the feature request can be taken seriously. Something like:

Section 1. Tables of O/S & filesystem Rules Describing legal & illegal filenames per O/S(-version) and per filesystem(-version).

Section 2. An Encoding/Decoding Algorithm pair To be used when writing to an filesystem which will reject the original name

Section 3. Description of one or more Rulesets for when Encoding will be applied for example Ruleset 1: Never Ruleset 2: Whenever “Needed”

Section 4. Statement of Rules for when Decoding will be applied for example Ruleset 1: Never Ruleset 2: Whenever the Database says an Encoding has been applied and we are syncing to an OS/fs that doesn’t require the encoding Ruleset 3: Whenever a (to be defined) Heuristic says an Encoding has been applied and we are syncing to an OS/fs that doesn’t require the encoding

Section 5. A proof that integrity can be maintained, and under what conditions

Section 5. Proposal for the UI for users to choose their options

@endolith Not really. You can use other tools to get back to the source device and rename the files. SSH or your feet work fine. Marking them invalid and refuse to sync should be default behavior. Option to rename and sync anyway may be reasonable, but disabled by default to avoid damage it may cause.