rclone: rclone sync complains of duplicate objects in destination, ignores and fails to remove them
What is the problem you are having with rclone?
rclone sync <localpath> remote:path sometimes gives errors about duplicate objects:
2020/07/12 02:54:56 NOTICE: 02 - Pixies - Tame.flac: Duplicate object found in destination - ignoring
2020/07/12 02:54:56 NOTICE: 03 - Pixies - Wave of Mutilation.flac: Duplicate object found in destination - ignoring
Successive runs of sync continue to emit these errors, and the duplicates are never cleaned up/dealt with.
Note that I’m concerned primarily with the fact that rclone isn’t cleaning up the dupes when sync is specified. So far I’m unclear about how the dupes occurred in the first place (probably from a run that received errors), so this ticket is only about not dealing with them once they exist.
What is your rclone version (output from rclone version)
rclone v1.52.2-DEV
- os/arch: linux/amd64
- go version: go1.11.6
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Debian Buster amd64
Which cloud storage system are you using? (eg Google Drive)
Opendrive with crypt overlay (standard)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone sync 'audio/Archives_FLAC/Pixies - Doolittle' 'odc:audio/Archives_FLAC/Pixies - Doolittle'
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
> rclone sync -vv 'audio/Archives_FLAC/Pixies - Doolittle' 'odc:audio/Archives_FLAC/Pixies - Doolittle'
2020/07/12 02:54:50 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "sync" "-vv" "audio/Archives_FLAC/Pixies - Doolittle" "odc:audio/Archives_FLAC/Pixies - Doolittle"]
2020/07/12 02:54:50 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2020/07/12 02:54:51 DEBUG : Starting OpenDrive session with ID: <hidden>
2020/07/12 02:54:53 DEBUG : Starting OpenDrive session with ID: <hidden>
2020/07/12 02:54:56 NOTICE: 02 - Pixies - Tame.flac: Duplicate object found in destination - ignoring
2020/07/12 02:54:56 NOTICE: 03 - Pixies - Wave of Mutilation.flac: Duplicate object found in destination - ignoring
2020/07/12 02:54:56 DEBUG : 02 - Pixies - Tame.flac: Size and modification time the same (differ by -100.856939ms, within tolerance 1s)
2020/07/12 02:54:56 DEBUG : 02 - Pixies - Tame.flac: Unchanged skipping
2020/07/12 02:54:56 DEBUG : 06 - Pixies - Dead.flac: Size and modification time the same (differ by -192.857485ms, within tolerance 1s)
2020/07/12 02:54:56 DEBUG : 06 - Pixies - Dead.flac: Unchanged skipping
...
2020/07/12 02:54:56 DEBUG : Encrypted drive 'odc:audio/Archives_FLAC/Pixies - Doolittle': Waiting for transfers to finish
2020/07/12 02:54:56 DEBUG : Waiting for deletions to finish
2020/07/12 02:54:56 INFO : There was nothing to transfer
2020/07/12 02:54:56 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 17 / 17, 100%
Elapsed time: 0.0s
2020/07/12 02:54:56 DEBUG : 3 go routines active
Subsequent rclone ls command shows the duplicates are still present:
> rclone ls 'odc:audio/Archives_FLAC/Pixies - Doolittle'
19952985 01 - Pixies - Debaser.flac
13447621 02 - Pixies - Tame.flac
13447621 02 - Pixies - Tame.flac
14458628 03 - Pixies - Wave of Mutilation.flac
14458628 03 - Pixies - Wave of Mutilation.flac
17164063 04 - Pixies - I Bleed.flac
23199378 05 - Pixies - Here Comes Your Man.flac
13583485 06 - Pixies - Dead.flac
19543610 07 - Pixies - Monkey Gone to Heaven.flac
12500938 08 - Pixies - Mr. Grieves.flac
9910476 09 - Pixies - Crackity Jones.flac
19199155 10 - Pixies - La La Love You.flac
25458291 11 - Pixies - No. 13 Baby.flac
11818476 12 - Pixies - There Goes My Gun.flac
20294560 13 - Pixies - Hey.flac
13222690 14 - Pixies - Silver.flac
18581304 15 - Pixies - Gouge Away.flac
2944387 Cover_Front_-_Large.jpg
11 notes.txt
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (9 by maintainers)
Commits related to this issue
- docs: Updated sync and dedupe command docs #4429 — committed to eharris/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to eharris/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to eharris/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to eharris/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to eharris/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to eharris/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to rclone/rclone by eharris 4 years ago
- docs: Updated sync and dedupe command docs #4429 — committed to negative0/rclone by eharris 4 years ago
I don’t want sync to do anything with dupes are normal operations don’t generate them.
I want to report that as an error and let me choose how to deal with it as it overcomplicates sync and adds unnecessary overhead to it as there are now two places to manage dedupe instead of one.
Sync is already a destructive operation and creates challenges for folks if they aren’t familiar with it as adding another destructive piece to it feels bad.
Just my two cents as I have a different perception on how it should work.