rclone: Rclone mount will incorrectly handle data when used with borgbackup

@Cubox- I am using https://beta.rclone.org/v1.51.0-327-g4006345c-beta/ on amd64 on a 5.4 kernel. Borg 1.1.11.

My remote is a google drive or a wasabi S3 bucket (tested both).

> rclone mount -v --dir-cache-time 5m --vfs-read-chunk-size 8M --vfs-read-chunk-size 100M --vfs-cache-mode writes --vfs-cache-max-size 4G --vfs-cache-max-age 6h drive-1:backup /mnt/temp
> # (different shell)

> borg init -e repokey /mnt/temp/test.borg
Enter new passphrase: 
Enter same passphrase again: 
Do you want your passphrase to be displayed for verification? [yN]: 
Failed to securely erase old repository config file (hardlinks not supported>). Old repokey data, if any, might persist on physical storage.

By default repositories initialized with this version will produce security
errors if written to with an older version (up to and including Borg 1.0.8).

If you want to use these older versions, you can disable the check by running:
borg upgrade --disable-tam /mnt/temp/test.borg

See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
Use "borg key export" to export the key, optionally in printable format.
Write down the passphrase. Store both at safe place(s).

# Recuding the segment size from 512mb to 100mb. Improves the 'reaction time' of borg when saving or trying to cancel a backup.
# My educated guess for a recommodation to not skrew things up:  `32MB <= (100MB per expected TB total repo size) <= 512MB`
# If you have fast upload (>=50Mbit), the segment size can be left larger
> sed -i "s/max_segment_size = 524288000/max_segment_size = 104857600/" /mnt/temp/test.borg/config

> borg create --progress --stats -C zstd /mnt/temp/test.borg::test1 /usr/src
Enter passphrase for key /mnt/temp/test.borg:
…
------------------------------------------------------------------------------                                                                                                     
Archive name: test1
Archive fingerprint: beeaaa3993ba8ba4e1bfd423941a2aafd4ee3fd6d8f511b1003aaaa05dda2b3c
Time (start): Sun, 2020-05-17 19:26:10
Time (end):   Sun, 2020-05-17 19:28:06
Duration: 1 minutes 55.85 seconds
Number of files: 137000
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                1.81 GB            431.30 MB            231.50 MB
All archives:                1.81 GB            431.30 MB            231.50 MB

                       Unique chunks         Total chunks
Chunk index:                   66626               131642
------------------------------------------------------------------------------

Unmounted, mounted again and tried an extract, worked without issues. Extracting took ~50 seconds.

Maybe you could post your versions and also try running rclone mount with -vv and post the output you get when trying borg init.

All my tests work fine so far!

Just chiming in to see what the latest is on this issue before I start a borg + rclone mount with vfs cache mode full backup strategy

I’m doing that, I have on my unraid rclone running with mount to Google Drive, I use this rclone mount for my borg backup repository. I found this thread looking for other people’s config, same as this one from borgbackup. In my case, it is working perfectly, no problems so far, I have rclone mount with this config: rclone mount --cache-dir /mnt/disks/ud1/RClone_Caching_Folder/ --vfs-cache-mode full --max-read-ahead 1024k --allow-other encrypted: /mnt/disks/secure & and then I point my borg repository inside the /mnt/disks/secure folder, so far running borg create works with no issues. This is what I run if it helps borg create --compression none --one-file-system --read-special --files-cache mtime,size --debug --show-rc --progress /ua_nvme1/testrepo::testrepo-{now} /mnt/user/repo1

rclone v1.55.0 borg 1.1.15

What @dragetd explains about reducing the segment size of borgbackup is something I need to try, that way the download from the cloud (in this case with rclone mount) would not take so long, because that’s exactly what it is happening to me, and I’m backing up about 500GB.

The work from RClone developers and Borg developers is amazing, I definitely need to donate something to those guys!

Great work and thank you @ncw!

@anaxonda add --allow-other to your rclone mount command. (Check it is allowed in /etc/fuse.conf first)

@segator This option is already available with the VFS rewrite from version 1.53 onwards, specifically the --vfs-write-back option