rclone: union: doesn't support --fast-list which slows down vfs/refresh
What is the problem you are having with rclone?
Doing a vfs/refresh remote command over a union with a remote take much time than over the remote.
I tried with the code but it’s too much golang for my level 😅. Could it be that it is not using the -fast-list option with the remote through the union mount point? Maybe it’s just a not covered case.
I notice this because I was getting timeouts with large folders if I mount the union instead the remote.
What is your rclone version (output from rclone version)
rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux 4.20.8-arch1-1-ARCH x86_64
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
With the config:
[gdrive]
type = drive
client_id = ...
client_secret = ...
token = ...
[union]
type = union
remotes = /srv/data gdrive:/data
And the remote mounted with remote control:
rclone mount union: /mnt/data-union \
--allow-other --allow-non-empty \
--buffer-size 512M \
--dir-cache-time 48h \
--fast-list \
--use-mmap \
--read-only \
--rc \
--vfs-read-chunk-size 8M \
--vfs-read-chunk-size-limit 256M
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
I put just the response times because the action it’s going well in any case.
time rclone rc vfs/refresh recursive=true dir=/path
{
"result": {
"/path": "OK"
}
}
real 0m30.601s
user 0m0.008s
sys 0m0.014s
But if I mount the gdrive:/data instead the union:
time rclone rc vfs/refresh recursive=true dir=/path
{
"result": {
"/path": "OK"
}
}
real 0m2.438s
user 0m0.007s
sys 0m0.011s
Thank you for the time in advance and sorry if something is wrong with the report.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 20 (12 by maintainers)
I’ve got the same response times with
fast_list = truein the config file 😢 .Great news! Thank you all!