rclone: webdav: nextcloud: rclone incorrectly retries long server-side chunks merge operations before they complete

What is the problem you are having with rclone?

Uploading very large files (e.g. 20 GB) always fails because rclone keeps restarting the transfer while the server is still under the process of merging the chunks together.

What is your rclone version?

❯ rclone --version
rclone v1.63.0
- os/version: darwin 13.3.1 (64 bit)
- os/kernel: 22.4.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.20.5
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits?

❯ neofetch
                    'c.          paul@Pauls-Air
                 ,xNMM.          --------------
               .OMMMMo           OS: macOS 13.3.1 22E261 arm64
               OMMM0,            Host: MacBookAir10,1
     .;loddo:' loolloddol;.      Kernel: 22.4.0
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 12 hours, 32 mins
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 331 (brew)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.9
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 1920x1080
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: yabai
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    Terminal: iTerm2
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal Font: MesloLGS-NF-Regular 13
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Apple M1
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Apple M1
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 2991MiB / 16384MiB
       .cooc,.    .,coo:.

Which cloud storage system are you using?

WebDAV (NextCloud)

The command you were trying to run?

rclone copy verybigfile.bin nextcloud:

A log from the command with the -vv flag

My NextCloud instance is running out of storage so unfortunately I’m not able to provide such a log. It has been a long-standing issue though.

@ncw I remember looking into it a very long time ago.

• For long operations, the HTTP MOVE request for merging chunks eventually times out (but the merge is actually still in progress).
• When trying to merge again then NextCloud returns a LOCKED error which I tried to ignore.
• rclone seems to somehow figure out that the file isn’t actually there, deletes the chunks, reuploads them, and tries to merge them again.
• Rinse and repeat.

There is no API to check whether the MOVE operation is still in progress or not. The only way I know of is to check if it’s still in progress is to retry the MOVE until we don’t receive a LOCKED error any longer.

If I recall correctly, this workaround didn’t work out: rclone eventually restarted the transfer anyway. Maybe rclone has some kind of mechanism to force retry “staled” transfers even if the backend asks it to wait?

👉 Would there be a way to work around this NextCloud limitation?

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don’t comment if you have no relevant information to add. It’s just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.