cgofuse: Slow performance writing compared to bazil.org/fuse

Hi!

I’m trying to track down a performance issue with cgofuse vs bazil.org/fuse in rclone.

Here is cgofuse mounting a local disk on Linux.

$ dd if=/tmp/1G of=/mnt/tmp/1G bs=128k
8192+0 records in
8192+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.2636 s, 58.8 MB/s

And here is the mount command. Note the 4k writes

rclone cmount -vv /tmp/data /mnt/tmp/ 2>&1 | grep "Write: "
...
2021/03/24 16:13:04 DEBUG : /1G: >Write: n=4096

And here is bazil.org/fuse

$ time dd if=/tmp/1G of=/mnt/tmp/1G bs=128k
8192+0 records in
8192+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.67968 s, 229 MB/s

And its mount command - note the 128k blocks

rclone mount -vv /tmp/data /mnt/tmp/ 2>&1 | grep "Write: "
...
2021/03/24 16:14:42 DEBUG : &{1G (w)}: >Write: written=131072, err=<nil>

I’d say this difference is entirely down to the different sized write blocks, but I haven’t been able to change it. I’ve tried loads of fuse parameters (max_write,auto_cache) and I haven’t been able to budge cgofuse from its 4k blocks for write.

Fuse mysteriously says this in the docs for max_write

max_write=N

Set the maximum number of bytes in a single write operation. The default is 128kbytes. Note, that due to various limitations, the size of write requests can be much smaller (4kbytes). This limitation will be removed in the future.

This was originally noted with macOS on the rclone forum but it replicated with linux for me very easily.

Any help much appreciated - running short of hair to pull out 😉

(Testing done with rclone master (built with go build -tags cmount to include cgofuse support on Linux, and use --debug-fuse to see the fuse debug).)

About this issue

Original URL
State: open
Created 3 years ago
Comments: 30 (8 by maintainers)

Most upvoted comments

It looks like it is something that has to be done sooner or later, although I am rather busy at the moment.

billziss-gh on Feb 12, 2023

libfuse2 is deprecated and does not support the feature. It’s only in libfuse3. And “fuse_msg_size” is a mergerfs thing. Not a libfuse thing.

https://github.com/libfuse/libfuse/commit/027d0d17c8a4605109f09d9c988e255b64a2c19a

trapexit on Mar 30, 2021

how does this perform on macOS

It’s the same 16k with or without the flag, so no difference. Setting blocksize to 1048576 does increase it to 128k-sized writes but causes file corruption.

darthShadow on Feb 20, 2023

max_pages impacts the max size of a fuse message. Practically speaking it impacts reads and writes the most (as getdent/readdir is currently limited to 4k messages by the kernel unfortunately.) Besides needing larger buffers to handle the size change there is no other effect.

trapexit on Mar 25, 2021

Do we want to set blocksize for macOS too? I am not sure if there are any downsides to it since there is almost 0 documentation about it apart from a mention in osxfuse/osxfuse#507 (comment)

The OSXFUSE mount options document mentions iosize, but not blocksize. We may have to consult the osxfuse/macfuse source on this.

Any thoughts on adding max-pages support too?

I am not opposed to add more options, but we must do so with a risk vs reward consideration. Every time we change something there is a risk that we will break an existing user. Our reward is of course that we fix a functionality or performance problem.

Unfortunately a lot of these options are not well described. Reading the source sometimes answers questions, but often real-world effects are missed. For example, does max_pages only affect size of writes as discussed here? Or can it have other consequences that may be deleterious in some scenarios?

Having said this I think there is a lot of value in what you are doing @darthShadow. If you or someone else compiles a comprehensive list of options/configurations/etc. that positively affect performance, I would certainly be interested to incorporate them into cgofuse in some form.

billziss-gh on Mar 25, 2021

Adding -o big_writes gives the same improvement as direct_io too. And that is enabled by default in bazil/fuse https://github.com/bazil/fuse/blob/master/fuse.go#L229

darthShadow on Mar 25, 2021

It actually seems correct based on my usage of it in mergerfs too across multiple servers to get the best performance. The moment we introduce kernel caching (by not adding direct_io), the performance drops down as it essentially double-caches the file with no control over how fast the kernel sends it to the application. Admittedly, I don’t know much about the internals of fuse, other than what I have learnt from my testing, so there may be some other explanation too.

Without direct_io:

darthshadow@server:~/test$ rclone cmount temp-1 temp-2 -vv --debug-fuse 2>&1 | grep Write: > cmount_no_direct_io_writes.txt
^C
darthshadow@server:~/test$ head -10 cmount_no_direct_io_writes.txt
2021/03/25 04:47:30 DEBUG : /1G.img: Write: ofst=0, fh=0x0
2021/03/25 04:47:30 DEBUG : /1G.img: >Write: n=4096
2021/03/25 04:47:30 DEBUG : /1G.img: Write: ofst=4096, fh=0x0
2021/03/25 04:47:30 DEBUG : /1G.img: >Write: n=4096
2021/03/25 04:47:30 DEBUG : /1G.img: Write: ofst=8192, fh=0x0
2021/03/25 04:47:30 DEBUG : /1G.img: >Write: n=4096
2021/03/25 04:47:30 DEBUG : /1G.img: Write: ofst=12288, fh=0x0
2021/03/25 04:47:30 DEBUG : /1G.img: >Write: n=4096
2021/03/25 04:47:30 DEBUG : /1G.img: Write: ofst=16384, fh=0x0
2021/03/25 04:47:30 DEBUG : /1G.img: >Write: n=4096

darthshadow@server:~/test$ (rm temp-2/1G.img || true) && dd if=1G.img of=temp-2/1G.img count=1024 bs=1048576
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 32.9387 s, 32.6 MB/s

With direct_io:

darthshadow@server:~/test$ rclone cmount -o direct_io temp-1 temp-2 -vv --debug-fuse 2>&1 | grep Write: > cmount_direct_io_writes.txt
^C
darthshadow@server:~/test$ head -10 cmount_direct_io_writes.txt
2021/03/25 04:46:59 DEBUG : /1G.img: Write: ofst=0, fh=0x0
2021/03/25 04:46:59 DEBUG : /1G.img: >Write: n=131072
2021/03/25 04:46:59 DEBUG : /1G.img: Write: ofst=131072, fh=0x0
2021/03/25 04:46:59 DEBUG : /1G.img: >Write: n=131072
2021/03/25 04:46:59 DEBUG : /1G.img: Write: ofst=262144, fh=0x0
2021/03/25 04:46:59 DEBUG : /1G.img: >Write: n=131072
2021/03/25 04:46:59 DEBUG : /1G.img: Write: ofst=393216, fh=0x0
2021/03/25 04:46:59 DEBUG : /1G.img: >Write: n=131072
2021/03/25 04:46:59 DEBUG : /1G.img: Write: ofst=524288, fh=0x0
2021/03/25 04:46:59 DEBUG : /1G.img: >Write: n=131072

darthshadow@server:~/test$ (rm temp-2/1G.img || true) && dd if=1G.img of=temp-2/1G.img count=1024 bs=1048576
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.29815 s, 147 MB/s

darthShadow on Mar 25, 2021