rclone: Move files using SFTP protocol is stucked

The associated forum post URL from `https://forum.rclone.org`

What is the problem you are having with rclone?

I tried to move items from remote using SFTP config, to local (Linux, Debian)

What is your rclone version (output from `rclone version`)

v1.56.2

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Debian 10 x64

Which cloud storage system are you using? (e.g. Google Drive)

SFTP

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

rclone move "my-sftp:mydata/" "/tolocal/" --verbose --progress --delete-empty-src-dirs --transfers 8

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

It is stucked in transfer files and is not 100% complete. If stop process and run again move command then there’s no other things to move and finish immediately.

As you see, those transfered items are not removed from queue when done and other remaining items won;t be moved because of full queue list.

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don’t comment if you have no relevant information to add. It’s just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (8 by maintainers)

Commits related to this issue

march: Fix unbounded parallelism when using --no-traverse When using `--no-traverse` the march routines call NewObject on each potential object in the destination. The concurrency limiter was arrang... — committed to rclone/rclone by ncw 9 months ago

Most upvoted comments

I don’t think so according to the backtrace. The SFTP backend periodically closes unused connections.

53 is quite a lot of connections though more than I would expect for --transfers 20 --checkers 20. I’d expect 40 or (maybe a few more). This doesn’t include the connections in use either which there are at least 20.

After a lot of investigation I discovered that --no-traverse can do --checkers * --checkers stat calls in parallel which is what is blowing up the number of connections. I fixed that to only do --checkers in parallel and the number of connections looks sensible now.

I think this will fix your issue as it will stop rclone overloading the server with too many connections to the point where it is rate limiting or refusing the connections.

Please give this a go!

v1.65.0-beta.7426.b17151815.fix-5824-sftp-connections on branch fix-5824-sftp-connections (uploaded in 15-30 mins)

ncw on Oct 10, 2023