rclone: Rclone sync fails - request wrongly url encoded?

Continuation of: https://forum.rclone.org/t/rclone-sync-fails-request-wrongly-url-encoded/23152

What is the problem you are having with rclone?

rclone sync fails with the error Couldn’t delete: Container Not Found Other commands work correctly, even rclone deletefile when deleting the file that failed during rclone sync

I have an analysis of what seems to be going wrong, the rclone sync command seems to be url encoding the request that fails. When I manually execute the request using curl, the url encoded variant returns a 404 - Not found.

If I manually decode the request then the response is as you would expect.

Request that fails:

HTTP REQUEST (req 0xc0024db300) 2021/03/25 16:17:13 DEBUG : GET /production%255Fencrypted%255Fsegments?format=json&limit=1000&prefix=mariadbservers%2Fweekly%2Fpiwik%255F2021%252D02%252D10.gz.bin%2F1612992682.03079176%2F9221554274 HTTP/1.1 Same request decoded, succeeds.

From Nick Craig-Wood: What it looks like is that rclone failed to URL decode the Manifest in the previous response

X-Object-Manifest: production%5Fencrypted%5Fsegments/mariadbservers/weekly/piwik%5F2021%2D02%2D10.gz.bin/1612992682.03079176/9221554274 And URL encoding that again produces the rather odd

GET /production%255Fencrypted%255Fsegments

What is your rclone version (output from rclone version)

I tried today’s latest beta.

Main version we are using though:

rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.7 LTS
Release:        16.04
Codename:       xenial

Which cloud storage system are you using? (e.g. Google Drive)

A combination of Openstack objectstore (Swift) and the encryption backend.

[backup]                                                                                                                                                                  
auth = XXX
key = XXXX
tenant = XXX
type = swift                                                                                                                                                                     
user = XXX

[backup_encrypted]                                                                                                                                                        
directory_name_encryption = false
filename_encryption = off
password = XXXXX
password2 = XXXX
remote = backup:/acceptance_encrypted/
type = crypt

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone sync /mnt/backup/mysql backup_encrypted:mariadbservers --transfers=20 --skip-links --stats 0

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

Logs for version master build 1.55: https://pastebin.com/pM11bBwS The relevant section in the log is the same for the latest beta.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@ncw I think I found the issue in the swift library: https://github.com/ncw/swift/pull/168