restic: Adding --ignore-inode to command line causes all files to be seen as "new"

I’ve poured through other issues, and the forum. Many similar discussions, but none that (seem to) involve --ignore-inode as the cause of files being seen as “new”. E.g.:

• Excessive file/disk reading despite available parent and identical metadata #2819
• restic 0.9.6 with new ctime behaviour causes all files to show as changed #2495

Output of restic version

restic 0.9.6 compiled with go1.13.4 on linux/amd64

How did you run restic exactly?

(Edit: As requested by @rawtaz, added more commands and their output, from the specific logged run in question, below. Using “✽✽✽” to mask user-specific information of debatable privacy.)

restic version
restic 0.9.6 compiled with go1.13.4 on linux/amd64

restic -r "${b2Bucket}" --cache-dir "/root/.cache/restic" list snapshots
20fc9ee03b5e771c9791fc95009a51bcbed2a2e1a801c8854a59448999c49eeb
221c8f81ee033997aa0b38d2f04befb5d061c4c3ca5e665325dd3b87d84bebfe
4ae00c9a0e0ffc3caa65bfde8bc79ce3a811c7e4f359048e6ff5914fef80a8e8
64ca94a813176ee662adf4d0fcf829935b16d2749943a47d6e976eb11295c4fc
8e0b1b6bc14527e400685d1bab98cb23a839256593c6b59837fb55f3e4ae53c3
90329c741c69fac7ff24521a9c48e39fc290fc5e09a208f5185360057d8db5ff
b70962d2a4e60b1849cb8bca6f1078e4e40fd15b3a5c2f279aec5392605a6857
bed6cd4f534ce46956269a1eee1e5943e71c17a8a72ea5d345d8ea79fac869fc
d198741a118698788096dec2ae9965aaa51a67009f22eaa78dfbacf350c474d2
ec74c0f9540fae9fda070b2c0f57d108fecee0f6e60c69c954ea8988a64da680
f5cf28357ba2e6355b02f1d74db6bab3b9dd3c9ebeb6d3bef896fdf014dbe79c
f8ec2c6d8dff9d4328e2c13b8aa0e108e9f49af4104977e69445252407b810d2

restic backup --verbose=2 -o b2.connections=8 -r ${b2Bucket}: --ignore-inode --exclude-caches --exclude-if-present .nobackup --cache-dir "/root/.cache/restic" --exclude-file "${excludeFilespec}"  "✽✽✽"
open repository
lock repository
load index files
start scan on [✽✽✽]
start backup on [✽✽✽]
new       ✽✽✽, saved in 0.149s (0 B added)
new       ✽✽✽, saved in 0.428s (0 B added)
...
...
...
new       ✽✽✽, saved in 0.148s (0 B added, 371 B metadata)

Files:       743386 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Data Blobs:     33 new
Tree Blobs:      5 new
Added to the repo: 10.539 MiB

processed 743386 files, 8.800 TiB in 26:41:50
snapshot 678804fb saved

Environment (verified at script runtime):

echo -e "${XDG_CACHE_HOME}\n${PWD}\n${USER}\n${HOME}"
/root/.cache/restic
/root
root
/root

The value of PATH is verified to be my normal complete system path, not the stripped-down one that you get from cron without some extra work.

There’s four differences between this command and environment, that now sees all files as “new” (but chunks do exist on server so nothing actually uploaded), and previous runs against existing data that has already been successfully backed up, which complete in mere minutes on a ~15TB volume with about 8TB to back up:

1. Added --ignore-inode because due to my specific requirements, I need to compare via mtime rather than ctime. (Being fully aware of the drawbacks of mtime.) I don’t actually need to ignore inode, it’s just apparently the only way to get restic to use mtime rather than ctime for now.
2. Explicitly specified --cache-dir, just in case the problem was that restic suddenly couldn’t find it for some reason.
3. Explicitly export XDG_CACHE_HOME, for the same reason, just in case (I’m sure not necessary), and running out of ideas.
4. Run via cron (with su - root as part of that for a fuller environment, same environment I got when successfully running restic after sudo -i). This was my eventual goal anyway but I tried it now in case that somehow solved it / hail-mary.

Nothing on the actual filesystem itself has changed. No additions, deletions, modifications, or metadata changes.

What backend/server/service did you use to store the repository?

Backblaze b2

Expected behavior

1. Perform a quick scan of the filesystem using new comparison method of --ignore-inode (I believe using path, mtime, size?) …
2. …against existing cached data that presumably has those attributes already cached.
3. See that still nothing has changed for any file.
4. Create new snapshot based 100% on existing chunks, without re-scanning any files, within <1 hour if not mere minutes.

Actual behavior

My backup script, exludes, and data set have finally settled down to statis. I’ve purposely not changed anything on the backup set, to get to this point. Now, every additional run completes in minutes, with nothing new scanned or uploaded.

But since adding the --ignore-inode flag, every file is reported as “new”, and fully re-scanned (which involves 15 HDDs grinding very hard for a very long time). Very little is uploaded - clearly not any new content - according to bmon, just a couple of KiB/s.

Do you have any idea what may have caused this?

I suspect - but can’t definitively prove without substantial effort and/or putting production data at risk - that adding the --ignore-inode was the cause. I’m afraid to try removing the flag - especially mid-way through the current run - due to 1) possibly causing some kind of cache corruption, and 2) I need to change the comparison method to mtime anyway, might as well bite the bullet and see what happens.

Do you have an idea how to solve the issue?

Probably not, but if this speculation is close enough, then:

• If changing the comparison --ignore-inode method is explicitly, intentionally eventually triggering full re-scans, then be smarter about when and why a rescan should be done, and don’t do it just because we’re using different comparison criteria.
• If mtime is not stored in the cache by default (without --ignore-inode), then it should be. The cost of doing so should be trivial in the big picture: an int64 to store, and you’re already inspecting file metadata anyway.

Did restic help you today? Did it make you happy in any way?

What I love about Restic:

• Vibrant, active, and professional developer and user community 0- with very little snark, passive-aggression, and/or abusiveness
• Written in Go
• My backup is more in my hands and can’t be arbitrarily revoked for disastrous business decisions (e.g. Crashplan).
• 130x faster backup speed than Crashplan (granted mainly due to cloud storage differences).

What I’m anxiously anticipating

• Faster prune.
  • My last prune took ~144 hours. 6 days! Yikes. That’s not remotely OK. Thankfully, it was the rare 6-day stretch without a sustained power outage or even so much as a momentary blip to internet connectivity. But my use case calls for at least every-other-month prunes.
  • I’m going to keep going with Restic for personal use in spite of this, because of the “what I love” above, but sadly, restic lost the competition for business backup, solely due to this.
  • I also run Borg Backup on an identical mirror, with pretty close to identical excludes. (But Borg regex vs Restic glob, means I’m not sure they are identical.) On a close to identical backup and prune schedule, Borg prunes about the same data with roughly the same rules, every time, in less than 5 minutes! (Granted, it’s to borgbase.com, which runs a Borg server. So maybe it has lots of server-side help. I appreciate Restic’s back-end flexibility, but minutes vs days is a no-brainer.)
• New --iexclude-file that ignores case, so I don’t have to make multiple case permutations of almost every exclude rule.
• Support regex in --exclude-file (or something like --exclude-regex-file).
• For the love of god, don’t treat --files-from entries as globbing! That is a baffling decision. No other software on earth does this. Not rsync, rclone…just about any other tool that deals with includes and excludes. They expressly expect a raw dump of individual explicit files to include, that was generated with some other tool. (“Do One Thing Well.”) Now we have to escape all problematic characters out of this list? (To be fair, Borg backup doesn’t even have this, not quite anyway - inexplicably. To me this seems like a “do not pass go” feature!)
  • If you don’t want to break the API, maybe add something like a new --files-raw-from flag. Or an environment variable to override the default behavior (e.g. RESTIC_FILESFROM_RAW).
• Faster rebuild-index.
  • Learning that I don’t actually need to do this after every backup interruption was a revelation.
  • That said, I still would like to do this about once a quarter. But at something like 4-5 days, that’s really sketchy. If a power or internet outage occurs during it, presumably my local cache is toast? And will never be good until a rebuild-index succeeds?