rclone: B2: "ERROR : : error reading destination directory: 200 calls/minute allowed for: API_LIST_FILE_NAMES_CALLED (429 too_many_requests)"

What is the problem you are having with rclone?

It is making too many calls per minute to the B2 API and then failing to recover gracefully from the resulting error.

What is your rclone version (output from rclone version)

rclone v1.47.0
- os/arch: linux/amd64
- go version: go1.12.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 64-bit

Which cloud storage system are you using? (eg Google Drive)

crypt remote backed by B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone check --download, can’t tell you the exact arguments since it’s run by a backup script not by me directly.

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Can’t reproduce on demand, so can’t give you a log output. This is the first time it has happened so unlikely to be able to reproduce frequently, but if I start seeing it more often I will modify the backup script to capture the verbose output.

Here’s the output I got:

2020/02/21 07:57:03 ERROR : : error reading destination directory: 200 calls/minute allowed for: API_LIST_FILE_NAMES_CALLED (429 too_many_requests)
2020/02/21 07:58:04 ERROR : : error reading destination directory: 200 calls/minute allowed for: API_LIST_FILE_NAMES_CALLED (429 too_many_requests)

I think rclone needs to be detecting this error and retrying after a suitable back-off time.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 29 (17 by maintainers)

Most upvoted comments

I’ve heard no reports of this recently.

I shall close this issue - if it turns up again we can re-open it.

Great news! Can you find the minimum value of Retry-After which works for you? It would be worth lowering from 5s if we can.

If not, 5s it is!

Do you want to send a pull request when you’ve found the minimum value?

Thanks

I was able to do a successful run of rclone with --low-level-retries 20. Thank you 👍

Does that means those files are skipped?

Yes it does. Rclone will sync what it can.

Try adding --fast-list does that help?