rclone: Large files uploaded via crypt remote to B2 are missing SHA1SUMs

What is your rclone version (eg output from rclone -V)

rclone v1.38
- os/arch: linux/amd64
- go version: go1.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Fedora 26 (Workstation Edition), 64 bit

Which cloud storage system are you using? (eg Google Drive)

B2 Backblaze

The command you were trying to run (eg rclone copy /tmp remote:tmp)

An example of a large file uploaded via crypt remote to B2 that misses SHA1SUM:

$ rclone mkdir b2:crypted-missing-sha1sum-test
$ rclone config
Configure new crypted remote named crypted-missing-sha1sum-test:

Remote config
--------------------
[crypted-missing-sha1sum-test]
remote = b2:crypted-missing-sha1sum-test
filename_encryption = off
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------

$ ls -lh ~/Dist/Fedora-Workstation-Live-x86_64-26-1.5.iso 
-rw-rw-r--. 1 tadej tadej 1.5G Jul  6 00:34 /home/tadej/Dist/Fedora-Workstation-Live-x86_64-26-1.5.iso
$ rclone copy -v ~/Dist/Fedora-Workstation-Live-x86_64-26-1.5.iso crypted-missing-sha1sum-test:
2017/10/22 15:06:20 INFO  : Encrypted drive 'crypted-missing-sha1sum-test:': Modify window is 1ms
2017/10/22 15:06:20 INFO  : Encrypted drive 'crypted-missing-sha1sum-test:': Waiting for checks to finish
2017/10/22 15:06:20 INFO  : Encrypted drive 'crypted-missing-sha1sum-test:': Waiting for transfers to finish
2017/10/22 15:07:20 INFO  : 
Transferred:   383.938 MBytes (6.041 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m3.5s
Transferring:
 *     Fedora-Workstation-Live-x86_64-26-1.5.iso: 25% done, 8.578 MBytes/s, ETA: 2m9s

[... output trimmed... ]

2017/10/22 15:23:43 INFO  : Fedora-Workstation-Live-x86_64-26-1.5.iso: Copied (new)
2017/10/22 15:23:43 INFO  : 
Transferred:   1.456 GBytes (1.424 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:    17m26.8s
$ rclone sha1sum b2:crypted-missing-sha1sum-test
                                          Fedora-Workstation-Live-x86_64-26-1.5.iso.bin
$ rclone sha1sum crypted-missing-sha1sum-test:
                                          Fedora-Workstation-Live-x86_64-26-1.5.iso

An example of the same large file uploaded directly to B2 remote that has SHA1SUM:

$ rclone mkdir b2:missing-sha1sum-test
$ ls -lh ~/Dist/Fedora-Workstation-Live-x86_64-26-1.5.iso 
-rw-rw-r--. 1 tadej tadej 1.5G Jul  6 00:34 /home/tadej/Dist/Fedora-Workstation-Live-x86_64-26-1.5.iso
$ rclone copy -v ~/Dist/Fedora-Workstation-Live-x86_64-26-1.5.iso b2:missing-sha1sum-test/
2017/10/22 13:37:22 INFO  : B2 bucket missing-sha1sum-test: Modify window is 1ms
2017/10/22 13:37:23 INFO  : B2 bucket missing-sha1sum-test: Waiting for checks to finish
2017/10/22 13:37:23 INFO  : B2 bucket missing-sha1sum-test: Waiting for transfers to finish
2017/10/22 13:37:37 INFO  : 
Transferred:   7.346 MBytes (467.081 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:       16.1s
Transferring:
 *     Fedora-Workstation-Live-x86_64-26-1.5.iso:  0% done, 1.299 MBytes/s, ETA: 19m2s

[... output trimmed... ]

2017/10/22 13:54:56 INFO  : Fedora-Workstation-Live-x86_64-26-1.5.iso: Copied (new)
2017/10/22 13:54:56 INFO  : 
Transferred:   1.456 GBytes (1.413 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:    17m35.3s

$ rclone sha1sum b2:missing-sha1sum-test
65880b7e61f995df0009ecf556d736552526d2e0  Fedora-Workstation-Live-x86_64-26-1.5.iso

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 28 (14 by maintainers)

Most upvoted comments

What is the largest file size I can upload to B2 via crypt without encountering this issue?

Up to the limit set by --b2-upload-cutoff you can set this up to 5G I think.

ncw on Nov 9, 2020

Yes this would be possible.

This is great news.

Yes, though I’d just use b2_copy_file as it doesn’t have any size limits.

I couldn’t find any size limits mentioned in b2_copy_part’s documentation, but I haven’t actually used this myself.

I think there may be one disadvantage to doing this. Doing a server side copy creates another version which doubles the storage. I suppose rclone could delete the source version once the copy is complete…

Yes, I agree with @ya-isakov, I think the source version should be deleted afterwards.

Maybe rclone should be doing this in the SetModTime call too?

I haven’t looked at the source code but if it only uses B2’s copy API to set a different modification time, then it would make sense to delete the original afterwards.

tjanez on Jun 7, 2019

@ncw, following #3210, I saw that you’ve implemented SetModTime using server side copy:

This creates a new version using a server side copy while updating the metadata.

Could we also leverage server side copy to set the SHA1 checksum after uploading the whole large file via the crypt remote?

Yes this would be possible.

As far as I understand it, we could do something like:

perform upload as we do it now

then create a server-side copy by:

specifying the correct SHA in the call to b2_start_large_file

repeating b2_copy_part

finishing with b2_finish_large_file

Yes, though I’d just use b2_copy_file as it doesn’t have any size limits.

I think there may be one disadvantage to doing this. Doing a server side copy creates another version which doubles the storage.

I suppose rclone could delete the source version once the copy is complete… Maybe rclone should be doing this in the SetModTime call too?

ncw on Jun 6, 2019

He said that they are aware of the gap in the protocol and he is going to discuss it with the engineers, so cross fingers they will implement a fix. I’ve always found Backblaze very helpful in this regard.

@ncw, thanks for pushing this.

I concur, it’s great to hear Backblaze is very helpful and willing to discuss extending their API to support such a use case. Fingers crossed!

tjanez on Nov 1, 2017