rclone: s3: rclone check and rclone copy s3->s3 slow and using too much memory

What is the problem you are having with rclone?

When I execute the rclone check command, I find that the execution of the path containing 5744 files is too slow, it seems to be stuck somewhere, and consumes 10+G of memory, the entire bucket size is more than 1 T, and check this folder There are more than 1000 subfolders under it.

What is your rclone version (output from rclone version)

rclone-v1.60.1

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

CentOS Linux release 7.5.1804 (Core)

Which cloud storage system are you using? (e.g. Google Drive)

Alibaba and Minio

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

./rclone check oss-source:gz-gov-open-doc/gz-old-site/attachment/201512/ minio-target:gz-gov-open-doc/gz-old-site/attachment/201512/ --missing-on-dst 201608.txt --checkers 64 -vv

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don’t comment if you have no relevant information to add. It’s just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.