asyncssh: Reading is slow when requesting more bytes than available
This may arguably be a questionable use case, but I noticed that read(1 << 30) (try to read 1 GiB from a 60 MB file) can be ~3x slower than read(-1). I would have expected both to be equally fast.
Consider this benchmark:
import random
import time
import numpy as np
import fsspec.implementations.sftp
import sshfs
hostname = "127.0.0.1"
port = 22
file_path = 'silesia.tar.gz'
fsspec_fs = fsspec.implementations.sftp.SFTPFileSystem(hostname, port=port)
sshfs_fs = sshfs.SSHFileSystem(hostname, port=port)
file_size = len(sshfs_fs.open(file_path).read())
print(f"Test file sized: {file_size} B")
for chunk_size_in_KiB in [-1, 4 << 20, 2 << 20, 1 << 20, 512 * 1024, 128 * 1024, 4 * 1024, 32]:
chunk_size = chunk_size_in_KiB * 1024 if chunk_size_in_KiB >= 0 else chunk_size_in_KiB
print(f"Try to read {chunk_size_in_KiB} KiB")
for open_file_name in ['fsspec_fs', 'sshfs_fs']:
file = globals()[open_file_name].open(file_path)
t0=time.time()
size = 0
for i in range((file_size + chunk_size - 1) // chunk_size if chunk_size > 0 else 1):
size += len(file.read(chunk_size))
t1=time.time()
assert size == file_size
file.close()
print(
f"Read {size / 1e6:.2f} MB in {chunk_size_in_KiB} KiB chunks with {open_file_name} "
f"in {t1-t0:.2f} s -> {size/(t1-t0)/1e6:.2f} MB/s"
)
Output:
Test file sized: 68238807 B
Try to read -1 KiB
Read 68.24 MB in -1 KiB chunks with fsspec_fs in 16.92 s -> 4.03 MB/s
Read 68.24 MB in -1 KiB chunks with sshfs_fs in 2.08 s -> 32.74 MB/s
Try to read 4194304 KiB
Read 68.24 MB in 4194304 KiB chunks with fsspec_fs in 50.17 s -> 1.36 MB/s
Read 68.24 MB in 4194304 KiB chunks with sshfs_fs in 25.15 s -> 2.71 MB/s
Try to read 2097152 KiB
Read 68.24 MB in 2097152 KiB chunks with fsspec_fs in 42.09 s -> 1.62 MB/s
Read 68.24 MB in 2097152 KiB chunks with sshfs_fs in 13.55 s -> 5.04 MB/s
Try to read 1048576 KiB
Read 68.24 MB in 1048576 KiB chunks with fsspec_fs in 42.13 s -> 1.62 MB/s
Read 68.24 MB in 1048576 KiB chunks with sshfs_fs in 7.37 s -> 9.26 MB/s
Try to read 524288 KiB
Read 68.24 MB in 524288 KiB chunks with fsspec_fs in 43.73 s -> 1.56 MB/s
Read 68.24 MB in 524288 KiB chunks with sshfs_fs in 4.67 s -> 14.62 MB/s
Try to read 131072 KiB
Read 68.24 MB in 131072 KiB chunks with fsspec_fs in 42.39 s -> 1.61 MB/s
Read 68.24 MB in 131072 KiB chunks with sshfs_fs in 2.37 s -> 28.78 MB/s
Try to read 4096 KiB
Read 68.24 MB in 4096 KiB chunks with fsspec_fs in 14.38 s -> 4.74 MB/s
Read 68.24 MB in 4096 KiB chunks with sshfs_fs in 2.03 s -> 33.63 MB/s
Try to read 32 KiB
Read 68.24 MB in 32 KiB chunks with fsspec_fs in 14.33 s -> 4.76 MB/s
Read 68.24 MB in 32 KiB chunks with sshfs_fs in 4.35 s -> 15.69 MB/s
So, trying to read 2 GiB, but only getting 60 MB, takes ~25 s, while it only takes 2 s with read(-1).
I do not know whether this is an issue with this wrapper or asyncssh directly because I was unable to adjust my (synchronously running) benchmark to use asnycssh directly.
I would guess that some code tries to do something O(size) for whatever reason even when it should iterate in chunks according to the buffer size, I’d think. I don’t see memory usage spiking, so at least nothing that large seems to get allocated.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (15 by maintainers)
Ok - I’ve implemented the OpenSSH “limits” request, and I am seeing some nice improvements when using it when no block size is set. Even with 128 parallel requests, the small block sizes really kill the through. So, increasing from my previous default of 16 kiB to OpenSSH’s default of 255 kiB is a big jump:
The last row here is with the OpenSSH max of 261120 bytes (255 kiB), picked up by making a “limits” request. This actually gives slightly better results than the larger read lengths, as the larger reads end up getting broken down into multiple 255 kiB requests when the server fails to respond with the full amount of data requested.
Since there no over-reading in these tests, the performance was about the same with or without the one-line fix I previously proposed to reduce the cost of over-reading. Fixing that properly will be a bit more complicated due to potential out-of-order read requests, though. These can be triggered when a client exceeds the server’s max read length, but it shouldn’t be too difficult to handle. I just need to keep track of the offset at the time EOF was returned and only allow reads earlier than that, rather than stopping all reads after EOF is seen.
I still need to do some testing and cleanup before checking this in, but it’s looking promising.
Regarding SCP, what you’re seeing there is not a change in request sizes. With SCP, you request a file and the server just sends the entire contents of the file to you in a single streamed response. There are multiple writes happening, but those writes are all coming from the server, without any additional requests being sent by the client. As such, the dynamic are different than SFTP. What you’re seeing is the result of the stream flow-control messages in SSH. By default, its default window size is 2 MiB. The client periodically acknowledges that it has read the data as it processes it, and this opens up the window for the server to send more. If a client ever stops reading or can’t keep up with the rate data is being sent, the window will eventually fill up and the server will block until the client can catch up. That’s not happening here, though – the adjustments are being made long before the window has filled up, so the server is basically pumping data as far as it can.
This fix is now available in AsyncSSH 2.18.0.
This change is now available as commit 46636d6 in the “develop” branch.