rclone: --bwlimit ignored for chunked uploads on b2 & drive (& others?)

When filing an issue, please include the following information if possible as well as a description of the problem. Make sure you test with the latest beta of rclone.

https://beta.rclone.org/
https://rclone.org/downloads/

If you’ve just got a question or aren’t sure if you’ve found a bug then please use the rclone forum instead of filing an issue.

What is your rclone version (eg output from rclone -V)

rclone v1.36

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 16.04 x64

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vv --bwlimit 350k --transfers 1 /path/to/files b2-crypt-remote:

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Transferred: 272.562 MBytes (258.207 kBytes/s) Errors: 0 Checks: 101 Transferred: 0 Elapsed time: 18m0.9s Transferring:

…large-file.ext: 1% done, 350.138 kBytes/s, ETA: 13h20m28s

2017/05/25 10:22:16 DEBUG : 405bo5jvqamhb26j3onqq0jj4k/io7s5pcc5bhbe7ag1dk09s5hpk/qt3tqkjokrm354a7vj755norv49hcb9215qe0o4jogsnbes50i2g/c52hpqov41uq2aehm0jg9tjm2pchh0bmfeia7bsgbg5lj7ppj8gg: Sending chunk 3 length 100663296 2017/05/25 10:22:16 DEBUG : 405bo5jvqamhb26j3onqq0jj4k/io7s5pcc5bhbe7ag1dk09s5hpk/qt3tqkjokrm354a7vj755norv49hcb9215qe0o4jogsnbes50i2g/c52hpqov41uq2aehm0jg9tjm2pchh0bmfeia7bsgbg5lj7ppj8gg: Didn’t find object to account part transfer

Even though rclone says it’s only using ~350k, several other bandwidth monitoring utilities show that it’s maxing out my upload speed, harming connectivity for other devices and services on my network. (350k is roughly half my upload speed which worked fine with Amazon Cloud Drive)

It’s also possible that it matches the bandwidth limit flag at some times, but not at others which may explain the discrepancy in the log. I tried to find an excerpt around the time I noticed issues,

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 19 (6 by maintainers)

Commits related to this issue

fs/accounting: rework to enable accounting to work with crypt and b2 This removes the old system of part accounting and replaces it with a system of popping off the accounting reader and wrapping up ... — committed to rclone/rclone by ncw 6 years ago

Most upvoted comments

I have fixed this in d0d6b83a7a9837dcd21b397bf76e2a641d127c5f which allows the backends to take the accounting/limiting off the incoming stream and place it back on the output of the buffers.

In my tests with b2 multipart uploads this smooths the bandwidth very well.

The drive and onedrive uploads have been fixed already.

I also fixed a similar issue in box and azureblob (which no-one had reported yet!)

One of the problems the old system had is that it didn’t work with crypt… This new one does!

https://beta.rclone.org/v1.39-120-g82418c30/ (uploaded in 15-30 mins)

ncw on Feb 2, 2018