rclone: [rclone v1.63.0] remote Nextcloud not working after upgrade from 1.62.2 to 1.63.0

then you need to set the chunksize to 0. according to the comments the /webdav endpoint does not implement chunking.

done²

I tested the v1.64.0 beta form your links above and confirm: It is working as it has worked up to 1.62.2 (with a Directory given in the webdav URL). So this would fix my initial bug.

Great.

Are you happy with this fix 705572d40027aec4d539f5b129f1fe87d1ab3f24 @devnoname120 ?

I’ve seen this usage before with webdav that people put subdirectories on the URL.

@devnoname120 wrote

Questions:

1. Does WebDAV actually support more finer-grained permissions than the user-level?

It does I believe but rclone doesn’t know about them.

1.1. If not, is there a strong point to support and maintain this use case?

Users are using it, so we should try not to break backwards compatibility!

1.2. Should we instead consider having some kind of subview remote type at the rclone level instead?

Yes, see the alias backend.

1. How does rclone approach this in other remotes? E.g. FTP, SCP, etc.

The ftp and sftp backends don’t allow this as they are configured with a host name rather than a URL. However the http backend does. A lot of the other backends allow this by setting the root_folder_id to a folder of the users choice also (eg drive).

If you use direct commands such as rclone copy, then I would advise against using this as a “line of defense” because the way that rclone interprets path operators .. and . may well change in the future.

Rclone is reasonably careful not to let .. escape up out of the configured remote. There was an issue about that some time ago that I remember fixing!

Overall, unless there is a WebDAV-specific behavior with subpaths (such as authentication, etc.) then I don’t think that it should be in the scope of backends to handle this properly. Depending on what we choose though, we may want to at least throw a warning explaining the reason why the URL is rejected.

What do you think?

I think that if it works OK we should allow it as it is currently in use and we try very hard not to break backwards compat!

I encourage you to modify the documentation and do a pull request on your own.

I agree that rclone should fallback to the previous no-chunk upload mechanism + display a warning instead of erroring out. See the 3rd comment above yours.

With regard to shadowdrive it’s their problem if they botched up Nextcloud’s APIs. We don’t support non-compliant Nextcloud setups. You should switch the remote type to plain WebDAV if the remote is not a properly-configured Nextcloud server.

On Oct 19, 2023, 13:43 +0200, Jan-Stefan Janetzky @.***>, wrote:

@darix wrote:

then you need to set the chunksize to 0. according to the comments the /webdav endpoint does not implement chunking. please add that to the docs over at https://rclone.org/webdav/ or even better, make it a default if it’s using /webdav… this new “feature” is literally breaking existing rclone configurations world wide. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

@Drakonas You are not supposed to manually mess up with the filesystem where NextCloud stores the user files. If you do that, then the filesystem cache of NextCloud doesn’t match the actual filesystem.

See here for a fix: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/occ_command.html#file-operations

With regard to the rclone move command at the start, could you retry it but with a sane filesystem cache? Your previous experiments may have left it in a bad state.

I see. Understood. I wondered if that was the case. That is unfortunate. I’ll make sure to do all operations over webdav then.

I also figured out the issue… Apparently the Music app currently prevents folders from being deleted due to a bug.

Also, can confirm… after doing files:scan and files:cleanup, move commands work fine with the latest beta. It found 0 errors, proably because I moved the data back to match what was in the cache before continuing. Apologies for the false alarm.

@ncw Thank you - I’ve done the testing: short summary: It works perfect!

More detailed: Tested the v1.64.0-beta as given in your post with a nextcloudremote with chunk_size 10M:

• big file (140M with internal chunking + merge) with webdav url with subdir => works
• small file (2M without internal chunking) with webdav url with subdir => works
• big file (140M with internal chunking + merge) with webdav url without subdir => works
• small file (2M without internal chunking) with webdav url without subdir => works

From my end this issue is fixed.

Thanks again to all people involved in developing rclone for your efforts to develop this wonderful utility and your help in fixing my issue!

@devnoname120 perfect! works! Tested with chunked upload (filesize > 10M) and without (filesize < 10M). Both worked successful.

Here is for completeness the debug output for the filesize > 10M: 2023/07/02 18:54:51 DEBUG : rclone: Version “v1.64.0-DEV” starting with parameters [“./rclone” “-vvv” “copy” “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip” “my-remotename:”] 2023/07/02 18:54:51 DEBUG : Creating backend with remote “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip” 2023/07/02 18:54:51 DEBUG : Using config file from “/root/.config/rclone/rclone.conf” 2023/07/02 18:54:51 DEBUG : fs cache: adding new entry for parent of “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip”, “/tmp” 2023/07/02 18:54:51 DEBUG : Creating backend with remote “my-remotename:” 2023/07/02 18:54:51 DEBUG : found headers: 2023/07/02 18:54:51 NOTICE: Chunks temporary upload directory: https://my-domainname/remote.php/dav/uploads/my-username/ 2023/07/02 18:54:51 DEBUG : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: Need to transfer - File not found at Destination 2023/07/02 18:54:51 DEBUG : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: Update will use the chunked upload strategy 2023/07/02 18:54:57 DEBUG : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: sha1 = afb7d8f00072c57db577d0a423ae9535d9d90d18 OK 2023/07/02 18:54:57 INFO : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: Copied (new) 2023/07/02 18:54:57 INFO : Transferred: 17.138 MiB / 17.138 MiB, 100%, 2.856 MiB/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 6.3s

2023/07/02 18:54:57 DEBUG : 7 go routines active

@mpsd Hmmm I see. Looks like we need to remove the additional path when building the temporary chunks upload directory.

@devnoname120 Here we go - internal chunking + merge does NOT work with url incl. dirpath / works with url without dirpath. Remote nextcloud configured as give per your advice (chunk_size: 10M). Both debug messages here:

1.64 beta with dirpath in nextcloud remote - NOT SUCESSFUL

2023/07/02 15:02:05 DEBUG : rclone: Version “v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked” starting with parameters [“rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64/rclone” “-vvv” “copy” “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64/rclone” “my-remote:”] 2023/07/02 15:02:05 DEBUG : Creating backend with remote “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64/rclone” 2023/07/02 15:02:05 DEBUG : Using config file from “/root/.config/rclone/rclone.conf” 2023/07/02 15:02:05 DEBUG : fs cache: adding new entry for parent of “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64/rclone”, “/tmp/rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64” 2023/07/02 15:02:05 DEBUG : Creating backend with remote “my-remote:” 2023/07/02 15:02:05 DEBUG : found headers: 2023/07/02 15:02:05 NOTICE: Chunks temporary upload directory: https://my-webdomain/remote.php/dav/uploads/my-username/rclone-w-path/ 2023/07/02 15:02:06 DEBUG : rclone: Need to transfer - File not found at Destination 2023/07/02 15:02:06 DEBUG : rclone: Update will use the chunked upload strategy 2023/07/02 15:02:06 ERROR : rclone: Failed to copy: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict 2023/07/02 15:02:06 ERROR : Attempt 1/3 failed with 1 errors and: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict 2023/07/02 15:02:06 DEBUG : rclone: Need to transfer - File not found at Destination 2023/07/02 15:02:06 DEBUG : rclone: Update will use the chunked upload strategy 2023/07/02 15:02:06 ERROR : rclone: Failed to copy: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict 2023/07/02 15:02:06 ERROR : Attempt 2/3 failed with 1 errors and: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict 2023/07/02 15:02:06 DEBUG : rclone: Need to transfer - File not found at Destination 2023/07/02 15:02:06 DEBUG : rclone: Update will use the chunked upload strategy 2023/07/02 15:02:06 ERROR : rclone: Failed to copy: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict 2023/07/02 15:02:06 ERROR : Attempt 3/3 failed with 1 errors and: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict 2023/07/02 15:02:06 INFO : Transferred: 0 B / 0 B, -, 0 B/s, ETA - Errors: 1 (retrying may help) Elapsed time: 0.7s

2023/07/02 15:02:06 DEBUG : 7 go routines active 2023/07/02 15:02:06 Failed to copy: chunked upload couldn’t purge upload directory: Sabre\DAV\Exception\Conflict: 409 Conflict

1.64 beta without dirpath in nextcloud remote - SUCESSFUL

2023/07/02 15:05:27 DEBUG : rclone: Version “v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked” starting with parameters [“rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64/rclone” “-vvv” “copy” “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip” “my-remote:/rclone-w-path/”] 2023/07/02 15:05:27 DEBUG : Creating backend with remote “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip” 2023/07/02 15:05:27 DEBUG : Using config file from “/root/.config/rclone/rclone.conf” 2023/07/02 15:05:27 DEBUG : fs cache: adding new entry for parent of “rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip”, “/tmp” 2023/07/02 15:05:27 DEBUG : Creating backend with remote “my-remote:/rclone-w-path/” 2023/07/02 15:05:27 DEBUG : found headers: 2023/07/02 15:05:27 NOTICE: Chunks temporary upload directory: https://my-webdomain/remote.php/dav/uploads/my-username/ 2023/07/02 15:05:27 DEBUG : fs cache: renaming cache item “my-remote:/rclone-w-path/” to be canonical “my-remote:rclone-w-path” 2023/07/02 15:05:27 DEBUG : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: Need to transfer - File not found at Destination 2023/07/02 15:05:27 DEBUG : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: Update will use the chunked upload strategy 2023/07/02 15:05:33 DEBUG : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: sha1 = afb7d8f00072c57db577d0a423ae9535d9d90d18 OK 2023/07/02 15:05:33 INFO : rclone-v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked-linux-amd64.zip: Copied (new) 2023/07/02 15:05:33 INFO : Transferred: 17.138 MiB / 17.138 MiB, 100%, 2.856 MiB/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 6.6s

2023/07/02 15:05:33 DEBUG : 7 go routines active

@ncw I tested the v1.64.0 beta form your links above and confirm: It is working as it has worked up to 1.62.2 (with a Directory given in the webdav URL). So this would fix my initial bug.

@devnoname120 my use cases are less driven by permission considerations but more of “can’t delete other data than in my current use case” by simply not seeing this data from the other use cases (or in other words: I can’t accidently wipe all data below root directory since I have not mapped root). I would prefer to continue to have this “line of defense” in the one time setting up the remote config but to have it in mind in every time I write or adjust a shell script or a one time shell command.

Thank you again for looking into this and helping me.

I tried relaxing the regexp checking the URL - please give this a go @mpsd

v1.64.0-beta.7121.705572d40.fix-7103-nextcloud-chunked on branch fix-7103-nextcloud-chunked (uploaded in 15-30 mins)

@devnoname120 @tcoupin you both had a hand in the development of the nextcloud chunking - does this seem sensible?