rclone: rclone fails to move small files to s3 buckets with default encryption enabled

Scripts to reproduce this bug can be found here: https://github.com/ccoakley/rclone-kms-s3-test

We wanted to make sure it wasn’t tied to our particular environment, so the above scripts can be run on a fresh aws account from an admin IAM user (or just follow along and do the steps manually). Note that a very current botocore is required to run the scripts, as this is a rather new feature.

We noticed that once we enabled default encryption on s3 buckets, small files failed to move with rclone. Once a file is large enough to transfer via multipart uploads, the problem goes away. Note that the etag for uploaded files is not stable (this can be seen in the debug output for the 3 retries). Tested with current beta.

To reproduce, create an s3 bucket and a kms key. Enable default encryption on the bucket using the new key. Use rclone to move a small (less than 5MB) file to the s3 bucket.

What is your rclone version (eg output from rclone -V)

rclone v1.38-095-g413faa99β

go version: go1.9.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

os/arch: darwin/amd64

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2017/11/14 13:22:55 DEBUG : Using config file from "/Users/ccoakley/development/rclone_bug/rclone.conf"
2017/11/14 13:22:55 DEBUG : rclone: Version "v1.38-095-g413faa99β" starting with parameters ["rclone" "-vv" "--config" "./rclone.conf" "move" "copy_small.txt" "s3:bucket_ec041f5862e945e8a41126f7c6f9256f"]
2017/11/14 13:22:55 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Modify window is 1s
2017/11/14 13:22:55 DEBUG : .git: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : .gitignore: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : cleanup.sh: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : exercise_bug.sh: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : finish_setup.py: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : rclone.conf: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : README.md: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : requirements.txt: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : settings.sh: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : setup.sh: Excluded from sync (and deletion)
2017/11/14 13:22:55 DEBUG : venv: Excluded from sync (and deletion)
2017/11/14 13:22:55 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Waiting for checks to finish
2017/11/14 13:22:55 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Waiting for transfers to finish
2017/11/14 13:22:55 ERROR : copy_small.txt: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "9bfb495fdac16b31bde2a790d7d98326"
2017/11/14 13:22:55 INFO  : copy_small.txt: Removing failed copy
2017/11/14 13:22:56 ERROR : copy_small.txt: Not deleting source as copy failed: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "9bfb495fdac16b31bde2a790d7d98326"
2017/11/14 13:22:56 DEBUG : Local file system at /Users/ccoakley/development/rclone_bug: Removing directory
2017/11/14 13:22:56 DEBUG : Local file system at /Users/ccoakley/development/rclone_bug: Failed to Rmdir: remove /Users/ccoakley/development/rclone_bug: directory not empty
2017/11/14 13:22:56 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "9bfb495fdac16b31bde2a790d7d98326"
2017/11/14 13:22:56 DEBUG : .git: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : .gitignore: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : cleanup.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : exercise_bug.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : finish_setup.py: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : rclone.conf: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : README.md: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : requirements.txt: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : settings.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : setup.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : venv: Excluded from sync (and deletion)
2017/11/14 13:22:56 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Waiting for checks to finish
2017/11/14 13:22:56 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Waiting for transfers to finish
2017/11/14 13:22:56 ERROR : copy_small.txt: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "0721aafabcc75dfd2e8477d00bf556e2"
2017/11/14 13:22:56 INFO  : copy_small.txt: Removing failed copy
2017/11/14 13:22:56 ERROR : copy_small.txt: Not deleting source as copy failed: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "0721aafabcc75dfd2e8477d00bf556e2"
2017/11/14 13:22:56 DEBUG : Local file system at /Users/ccoakley/development/rclone_bug: Removing directory
2017/11/14 13:22:56 DEBUG : Local file system at /Users/ccoakley/development/rclone_bug: Failed to Rmdir: remove /Users/ccoakley/development/rclone_bug: directory not empty
2017/11/14 13:22:56 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "0721aafabcc75dfd2e8477d00bf556e2"
2017/11/14 13:22:56 DEBUG : .git: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : .gitignore: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : cleanup.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : exercise_bug.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : finish_setup.py: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : rclone.conf: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : README.md: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : requirements.txt: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : settings.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : setup.sh: Excluded from sync (and deletion)
2017/11/14 13:22:56 DEBUG : venv: Excluded from sync (and deletion)
2017/11/14 13:22:56 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Waiting for checks to finish
2017/11/14 13:22:56 INFO  : S3 bucket bucket_ec041f5862e945e8a41126f7c6f9256f: Waiting for transfers to finish
2017/11/14 13:22:56 ERROR : copy_small.txt: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "05c0d3de94e16831b9922aba416018f4"
2017/11/14 13:22:56 INFO  : copy_small.txt: Removing failed copy
2017/11/14 13:22:57 ERROR : copy_small.txt: Not deleting source as copy failed: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "05c0d3de94e16831b9922aba416018f4"
2017/11/14 13:22:57 DEBUG : Local file system at /Users/ccoakley/development/rclone_bug: Removing directory
2017/11/14 13:22:57 DEBUG : Local file system at /Users/ccoakley/development/rclone_bug: Failed to Rmdir: remove /Users/ccoakley/development/rclone_bug: directory not empty
2017/11/14 13:22:57 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "05c0d3de94e16831b9922aba416018f4"
2017/11/14 13:22:57 Failed to move: corrupted on transfer: MD5 hash differ "d1a661a9218bd8c35ced78e3cc31db77" vs "05c0d3de94e16831b9922aba416018f4"

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 4
Comments: 21 (11 by maintainers)

Commits related to this issue

s3: document --ignore-checksum workaround for KMS #1824 — committed to rclone/rclone by ncw 6 years ago
s3: fix hashes on small files with aws:kms and sse-c If rclone is configured for server side encryption - either aws:kms or sse-c (but not sse-s3) then don't treat the ETags returned on objects as MD... — committed to rclone/rclone by ncw 4 years ago
s3: fix hashes on small files with aws:kms and sse-c If rclone is configured for server side encryption - either aws:kms or sse-c (but not sse-s3) then don't treat the ETags returned on objects as MD... — committed to rclone/rclone by ncw 4 years ago
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827 — committed to rclone/rclone by ncw 4 years ago
s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C Before this change, small objects uploaded with SSE-AWS/SSE-C would not have MD5 sums. This change adds metadata for these objects in the ... — committed to rclone/rclone by ncw 4 years ago
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827 — committed to rclone/rclone by ncw 4 years ago
s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C Before this change, small objects uploaded with SSE-AWS/SSE-C would not have MD5 sums. This change adds metadata for these objects in the ... — committed to rclone/rclone by ncw 4 years ago
s3: fix hashes on small files with aws:kms and sse-c If rclone is configured for server side encryption - either aws:kms or sse-c (but not sse-s3) then don't treat the ETags returned on objects as MD... — committed to scylladb/rclone by ncw 4 years ago
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827 — committed to scylladb/rclone by ncw 4 years ago
s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C Before this change, small objects uploaded with SSE-AWS/SSE-C would not have MD5 sums. This change adds metadata for these objects in the ... — committed to scylladb/rclone by ncw 4 years ago
s3: fix hashes on small files with aws:kms and sse-c If rclone is configured for server side encryption - either aws:kms or sse-c (but not sse-s3) then don't treat the ETags returned on objects as MD... — committed to scylladb/rclone by ncw 4 years ago
s3: store md5 in the Object rather than the ETag This enables us to set the md5 to cache it. See: #1824 #2827 — committed to scylladb/rclone by ncw 4 years ago
s3: Add MD5 metadata to objects uploaded with SSE-AWS/SSE-C Before this change, small objects uploaded with SSE-AWS/SSE-C would not have MD5 sums. This change adds metadata for these objects in the ... — committed to scylladb/rclone by ncw 4 years ago
If S3 KMS is enabled then MD5 corruption error * Workaround can be replaced after rclone v1.54 release with option server_side_encryption: aws:kms * https://github.com/rclone/rclone/issues/1824 — committed to odahu/odahu-flow by vlad-tokarev 4 years ago
If S3 KMS is enabled then MD5 corruption error (#448) * Workaround can be replaced after rclone v1.54 release with option server_side_encryption: aws:kms * https://github.com/rclone/rclone/issues/18... — committed to odahu/odahu-flow by vlad-tokarev 4 years ago

Most upvoted comments

I think what needs to be done is that if the config parameter server_side_encryption is set to aws:kms the rclone should just be ignoring the Etag and only using the one rclone provides.

Does anyone want to have a go at this?

ncw on Sep 2, 2018

My bad - that option is not exposed on the UI frontend I’m using.

evanthomas on Apr 18, 2023

@evanthomas have you told rclone you are using server side encryption?

See https://rclone.org/s3/#key-management-system-kms

Key Management System (KMS)

If you are using server-side encryption with KMS then you must make sure rclone is configured with server_side_encryption = aws:kms otherwise you will find you can’t transfer small objects - these will create checksum errors.

ncw on Apr 18, 2023

I’ve had a go at fixing this. Note that you will need server_side_encryption = aws:kms set in your config (or supplied as a flag).

Testing appreciated 😃

v1.54.0-beta.4905.cbd93519c.fix-s3-sse on branch fix-s3-sse (uploaded in 15-30 mins)

ncw on Nov 20, 2020

Then what does rclone use md5 hashes for?

For doing whole file data integrity checks.

ncw on Mar 29, 2020