rclone: union: doesn't support --fast-list which slows down vfs/refresh

What is the problem you are having with rclone?

Doing a vfs/refresh remote command over a union with a remote take much time than over the remote.

I tried with the code but it’s too much golang for my level 😅. Could it be that it is not using the -fast-list option with the remote through the union mount point? Maybe it’s just a not covered case.

I notice this because I was getting timeouts with large folders if I mount the union instead the remote.

What is your rclone version (output from rclone version)

rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 4.20.8-arch1-1-ARCH x86_64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

With the config:

[gdrive]
type = drive
client_id = ...
client_secret = ...
token = ...

[union]
type = union
remotes = /srv/data gdrive:/data

And the remote mounted with remote control:

rclone mount union: /mnt/data-union \
   --allow-other --allow-non-empty \
   --buffer-size 512M \
   --dir-cache-time 48h \
   --fast-list \
   --use-mmap \
   --read-only \
   --rc \
   --vfs-read-chunk-size 8M \
   --vfs-read-chunk-size-limit 256M

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

I put just the response times because the action it’s going well in any case.

time rclone rc vfs/refresh recursive=true dir=/path

{
	"result": {
		"/path": "OK"
	}
}

real	0m30.601s
user	0m0.008s
sys	0m0.014s

But if I mount the gdrive:/data instead the union:

time rclone rc vfs/refresh recursive=true dir=/path

{
	"result": {
		"/path": "OK"
	}
}

real	0m2.438s
user	0m0.007s
sys	0m0.011s

Thank you for the time in advance and sorry if something is wrong with the report.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 20 (12 by maintainers)

Most upvoted comments

I wonder if you could add

fast_list = true

to the config for the underlying remote in the rclone.conf and see a time difference as then it should use fast-list with the underlying remote that’s part of the union.

#661

I’ve got the same response times with fast_list = true in the config file 😢 .

Great news! Thank you all!