sftp: How to avoid upload speed degrading with latency?

We’ve been integrating sftp package into rclone and it is working very well - a very nicely designed package, thank you.

However a user has noticed a performance problem which I’ve confirmed - the more latency in the internet connection, the slower the transfers.

I’ve been using netelem to set delays on localhost and trying transfers to and from a local sftp server.

At 1ms delay I get 100 MB/s
At 10ms delay I get 73 MB/s
At 100ms delay I get 9 MB/s

rclone uses the File.ReadFrom method to upload data.

I tried increasing MaxPacket in the client using a delay of 100ms, but anything greater than the default 32k just seemed to make things slower.

Any ideas on how to speed things up much appreciated.

Cheers

Nick

FYI Here is how I tested with rclone to a local sftp server

$ sudo tc qdisc change dev lo root netem delay 1ms
$ rclone -v copy /home/files/ISOs/ubuntu-16.04.1-desktop-amd64.iso TestSftp:
2017/02/20 22:28:02 INFO  : 
Transferred:   1.409 GBytes (100.970 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       14.2s
$ rclone delete TestSftp:ubuntu-16.04.1-desktop-amd64.iso

$ sudo tc qdisc change dev lo root netem delay 10ms
$ rclone -v copy /home/files/ISOs/ubuntu-16.04.1-desktop-amd64.iso TestSftp:
2017/02/20 22:28:38 INFO  : 
Transferred:   1.409 GBytes (73.208 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       19.7s
$ rclone delete TestSftp:ubuntu-16.04.1-desktop-amd64.iso

$ sudo tc qdisc change dev lo root netem delay 100ms
$ rclone -v copy /home/files/ISOs/ubuntu-16.04.1-desktop-amd64.iso TestSftp:
2017/02/20 22:31:34 INFO  : 
Transferred:   1.409 GBytes (9.362 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:     2m34.1s
$ rclone delete TestSftp:ubuntu-16.04.1-desktop-amd64.iso

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 22 (18 by maintainers)

Commits related to this issue

Document that ReadFrom/WriteTo are best for high throughput - fixes #158 — committed to ncw/sftp by ncw 6 years ago
Merge pull request #250 from ncw/fix-158 Document that ReadFrom/WriteTo are best for high throughput fixes #158 — committed to pkg/sftp by eikenb 6 years ago

Most upvoted comments

I sat down to have a look at the source code to see how to implement my idea and a few minutes later I realised that this idea has already been implemented in the WriteTo function. The WriteTo function keeps the window open while reading from the SFTP file and writing to the client’s file.

https://github.com/pkg/sftp/blob/d2aa419835609d317c34baae11bc345d1f9971a7/client.go#L861-L863

So I adjusted rclone to use WriteTo. This involved a bit of messing about with io.Pipe to convert the io.Writer that you give to WriteTo back into an io.Reader which I need for the rclone internals. (I’ve done this before so it only took a few minutes).

And voila - the speed is now at the 19MB/s I was hoping for over a 100mS link. This is on a par with the sftp binary 😃

Doubtless this can be improved by tweaking buffers etc, but to have it on a par with sftp is good enough for the moment.

I’m already using the ReadFrom function to upload files with rclone and that works at about 19 MB/s too (I wish I’d noticed this before!).

So the lesson learned is use WriteTo and ReadFrom if you have links with long latency and wish to stream transfers quickly instead of the Read and Write methods.

It might be worth mentioning that in the docs - shall I prepare a PR?

ncw on May 24, 2018