restic: Inconsistent duration

Output of `restic version`

restic 0.7.1 compiled with go1.8.3 on linux/amd64

How did you run restic exactly?

restic backup -x --exclude-file /tmp/restic-exclude / restic backup -x --exclude-file /tmp/restic-exclude /home restic backup -x --exclude-file /tmp/restic-exclude /home /home/dottoreg/Private # now combine the backup of the above sources in one backup: restic backup -x --exclude-file /tmp/restic-exclude / /home /home/dottoreg/Private

What backend/server/service did you use?

File based to USB-attached hd. The path is stored in the environment variable (as is the password).

Expected behavior

I would expect a consistent amount and duration of the backup processes.

Actual behavior

using parent snapshot effbd27c
scan [/]
scanned 58009 directories, 605162 files in 0:22
[3:08] 100.00%  72.644 MiB/s  13.342 GiB / 13.342 GiB  663170 / 663171 items  0 errors  ETA 0:00 
duration: 3:08, 72.34MiB/s
snapshot 743e02cd saved

using parent snapshot effbd27c
scan [/home]
scanned 3923 directories, 96649 files in 0:14
[11:24] 100.00%  18.642 MiB/s  12.452 GiB / 12.452 GiB  100572 / 100572 items  0 errors  ETA 0:00 
duration: 11:24, 18.62MiB/s
snapshot 2cb14872 saved

using parent snapshot effbd27c
scan [/home/dottoreg/Private]
scanned 76749 directories, 133789 files in 12:18
[21:00] 100.00%  23.673 MiB/s  29.129 GiB / 29.128 GiB  210538 / 210538 items  0 errors  ETA 0:00 
duration: 21:00, 23.67MiB/s
snapshot b2b616d3 saved

The combined run takes much more time to scan, much more time for the backup. The Throughput is much lower:

using parent snapshot ba12eb4a
scan [/ /home /home/dottoreg/Private]
scanned 296102 directories, 1199820 files in 20:20
[1:17:59] 100.00%  27.483 MiB/s  125.581 GiB / 125.574 GiB  1495922 / 1495922 items  0 errors  ETA 0:00 
duration: 1:17:59, 27.48MiB/s
snapshot effbd27c saved

My problem: Sum of durations: 3:08 + 11:24 + 21:00 is much lower than 1:17:59. (It’s a factor of two!) Sum of scan times: 0:22 + 0:14 + 12:18 is much lower than 20:20.

Steps to reproduce the behavior

See above

Do you have any idea what may have caused this?

No.

Do you have an idea how to solve the issue?

Perhaps it is not the same when backing up the isolated items compared to one backup with all three items. Should I stay with three isolated backups?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 25 (13 by maintainers)

Most upvoted comments

In order to find something within the repo, we need to have an anchor point. In the current implementation, that’s the parent snapshot, and I agree the implementation is not the best. I’m already working on a new one.

If you want to understand how the data structures look like and what’s stored in the repo, I recommend having a look at the repo specification, here: https://restic.readthedocs.io/en/latest/100_references.html#design

fd0 on Nov 26, 2017