tokio: File operations don't use the whole buffer

tokio 0.2.4, see https://github.com/tokio-rs/tokio/blob/0d38936b35779b604770120da2e98560bbb6241f/tokio/src/io/blocking.rs#L29.

There’s an unexpected 16 KB limit for IO operations, e.g. this will print 16384. It seems intended, but it’s somewhat confusing.

async fn run() -> Result<(), std::io::Error> {
    let mut file = File::open("x.mkv").await?;
    let mut buf = [0u8; 32768];
    let size = file.read(&mut buf).await?;
    println!("{}", size);
    Ok(())
}

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 5
Comments: 21 (14 by maintainers)

Commits related to this issue

io: replace Vec<u8> for BytesMut in Buf As hinted in https://github.com/tokio-rs/tokio/issues/1976#issuecomment-568213794 this change replaces the inner buf attribute of the Buf struct. — committed to blasrodri/tokio by blasrodri 4 years ago
update to use 64KB msize, don't use tokio for files tokio issue: - https://github.com/tokio-rs/tokio/issues/1976 This means when we pass a buffer > 16KB to the OS, tokio truncates it to 16KB blowing... — committed to oxidecomputer/p9fs by rcgoodfellow 2 years ago

Most upvoted comments

Looking at your original issue, I think we can support better performance by working directly with Bytes. In that case, we can avoid the copying and instead send the bytes handle to the remote thread.

carllerche on Dec 21, 2019

I am open to changing the buffer size used by the File type.

Darksonn on Jan 22, 2022

It’s an extremely narrow test, but in my quest to optimize I/O performance on something reading/writing whole (1-50GiB) files sequentially, I tested a quick hack that simply changes MAX_BUF to 128MiB and, much to my surprise, it Just Worked™: With a 128MiB buffer, I’m getting 128MiB per read() and write() syscall according to strace.

This is an obnoxiously large I/O size, but it does work on this very specific use case: The files are on cephfs, in an erasure coded pool, with 128MiB object size. Aligning I/O to read whole objects per request substantially improves performance (approx 80MiB/s per task with four tasks up to approx. 220MiB/s in my case)

This is on Fedora with kernel 5.6.13

EDIT: Fixed numbers

mcronce on Jan 22, 2022

Well, the options are the following:

We change the default buffer size.
We provide a way to configure the buffer size.
You perform the operations yourself with spawn_blocking instead of going through tokio::fs::File.

I think I would be ok with all of these. What do you think?

Darksonn on Mar 21, 2022

@Darksonn is there anything we could help with to get this issue addressed?

At Deno we got several reports regarding poor performance when working on large files (https://github.com/denoland/deno/issues/10157) due to a need to perform thousands of roundtrips between Rust and JavaScript to read the data.

bartlomieju on Mar 21, 2022

The reason to have a maximum buffer size is that the file API allocates an intermediate buffer separate from the user-provided buffer. I don’t think the exact choice of size was benchmarked.

Darksonn on Jun 28, 2021