syncthing: Rename special/unsupported characters in filenames
As mentioned in https://github.com/syncthing/syncthing-android/issues/192 , some filenames are not accepted by windows hosts because they contain ‘special characters’ like colons or bars.
@schuft69 suggested to add a “Rename special characters to ‘_’” option to resolve this issue.
< (less than)
> (greater than)
: (colon)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)
About this issue
- Original URL
- State: open
- Created 12 years ago
- Reactions: 7
- Comments: 62 (19 by maintainers)
Commits related to this issue
- lib/fs: Add syncing of reserved FAT characters (fixes #1734) See https://github.com/syncthing/syncthing/issues/9539 for more details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters (fixes #1734) See https://github.com/syncthing/syncthing/issues/9539 for more details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters (fixes #1734) See https://github.com/syncthing/syncthing/issues/9539 for more details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters fixes #1734 fixes #9539 See #9539 for details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters fixes #1734 fixes #9539 See #9539 for details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters fixes #1734 fixes #9539 See #9539 for details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters fixes #1734 fixes #9539 See #9539 for details. — committed to rasa/syncthing by rasa 2 years ago
- lib/fs: Add syncing of reserved FAT characters fixes #1734 fixes #9539 See #9539 for details. fix typos, tweak comments, gci write Add metric to EncoderType fix tests (almost) fix(config): fix co... — committed to rasa/syncthing by rasa 2 years ago
These are now out of sync items, which explain why they are out of sync, so I think the problem is somewhat solved. You know what you need to do, and I’d rather not have dashes and random stuff added to my filenames by some software I am using.
I think it would be good if Syncthing could tackle the problem. It’s the kind of problem that needs to be tackled when building a sync solution for everyone that is easy to use. Most people would not mind if “:” is converted to “_” and those that do could change the default setting, whereas not syncing a file at all has bigger impact.
Dropbox replace trailing spaces in filenames on any platform, when first detected. If you create a file "test " with trailing space on Mac, then as soon as Dropbox detects it (even if you aren’t syncing with any other platforms), it will rename the file locally to “test” without the trailing space.
I am working on a new kind of sync app for Ronomon, and recently worked on replacing reserved characters with underscores.
These are the characters that would need to be replaced to support almost every platform:
These characters should be replaced with as many underscores.
And then these device names also need to be modified slightly because AUX and AUX.txt are invalid on Windows but AUX_.txt is fine:
These reserved device names should be automatically appended with an underscore (e.g. “AUX_” or “aux_.txt”).
Please let me know if something is left out here.
If it would be helpful here, then these are some key ideas which might make the problem manageable and not so hard:
This should cover:
exFAT (http://en.wikipedia.org/wiki/ExFAT) VFAT (http://en.wikipedia.org/wiki/File_Allocation_Table#VFAT) NTFS (http://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations) HFS+ (http://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations)
For non-case-preserving filesystems (FAT12, FAT16, FAT32), we may also need to replace “+,.;=[]!@” but I have not tested this yet.
Hope this helps.
Should this thread be closed, with a link to a new issue called “Write a Detailed Specification for How to Reliably Translate and Sync Filenames that are Illegal on Some Platforms”?
We can at least sketch out what that spec has to contain before the feature request can be taken seriously. Something like:
Section 1. Tables of O/S & filesystem Rules Describing legal & illegal filenames per O/S(-version) and per filesystem(-version).
Section 2. An Encoding/Decoding Algorithm pair To be used when writing to an filesystem which will reject the original name
Section 3. Description of one or more Rulesets for when Encoding will be applied for example Ruleset 1: Never Ruleset 2: Whenever “Needed”
Section 4. Statement of Rules for when Decoding will be applied for example Ruleset 1: Never Ruleset 2: Whenever the Database says an Encoding has been applied and we are syncing to an OS/fs that doesn’t require the encoding Ruleset 3: Whenever a (to be defined) Heuristic says an Encoding has been applied and we are syncing to an OS/fs that doesn’t require the encoding
Section 5. A proof that integrity can be maintained, and under what conditions
Section 5. Proposal for the UI for users to choose their options
@endolith Not really. You can use other tools to get back to the source device and rename the files. SSH or your feet work fine. Marking them invalid and refuse to sync should be default behavior. Option to rename and sync anyway may be reasonable, but disabled by default to avoid damage it may cause.