restic: Slow backup speed for large (incremental) backups
Hi,
I notice that restic doesn’t seem to be optimised for unchanged files which makes backing up very large repos (~500GBs for example with some large files) quite time consuming.
Not sure if this would work internally, but you must be keeping a map from file->chunks so can’t you notice that a file is the “same” (e.g. same hashcode, date/time access and file size?) and then simply clone the file->chunks map which should be significantly quicker?
I hesitate to raise this because having seen the calibre of this tool I can’t believe this hasn’t already occurred to you 😃.
Thanks!
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 26 (15 by maintainers)
That’s no problem, only members of the restic organisation can add labels, I’ll gladly do that for you (and remove redundant bits from the title).
Restic does indeed already check whether files have changed, but since we do not yet hold any local state, the information needs to be fetched from the repository. That’s what makes it slow at the moment. I’m planning to add a local metadata cache (file names, modification timestamps, lists of blobs etc.), that should improve the speed a lot!
Please note that interrupted backups are resumed already, at least concerning transferred data. Restic regularly creates checkpoints for new data that needed to be saved in the repo, so most data is not transferred twice even when the backup was interrupted.
I consider the slow speed for incremental (=not the first) backups a bug.
I can confirm this behaviour. I have an minio server setup and a rather large directory&file tree, ~6Gb.
Initial backup took 12 minutes I removed one file and now
restic backupETA is 13 hours!Is there anything we can do localy to solve the issue? Or, as of yet, restic is not ready for big directory trees?