restic: Compression leads to content mismatch after restore!
Output of restic version
This one works well:
/root/restic version
restic 0.13.1 compiled with go1.18 on linux/amd64
This one is current master compiled for compression tests:
/root/restic2 version
restic 0.13.1-dev (compiled manually) compiled with go1.18 on linux/amd64
How did you run restic exactly?
Simple backup with content comparing after restore.
source file:
dd if=/dev/sde of=/mnt/img1 bs=1M count=128
Current master WITH compression:
backup:
cat /mnt/img1 | /root/restic2 -r rest:http://dc15s3:8000/repo2/ --verbose backup --compression auto --stdin --stdin-filename img1 --password-file ./pass
open repository
repository 193651e2 opened (repo version 2) successfully, password is correct
lock repository
load index files
read data from stdin
start scan on [/img1]
start backup on [/img1]
scan finished in 2.952s: 1 files, 0 B
Files: 1 new, 0 changed, 0 unmodified
Dirs: 0 new, 0 changed, 0 unmodified
Data Blobs: 87 new
Tree Blobs: 1 new
Added to the repo: 128.006 MiB
processed 1 files, 128.000 MiB in 0:12
snapshot b727ac47 saved
/root/restic2 -r rest:http://dc15s3:8000/repo2/ --password-file ./pass snapshots
repository 193651e2 opened (repo version 2) successfully, password is correct
ID Time Host Tags Paths
------------------------------------------------------------
b727ac47 2022-06-03 15:04:38 dc15 /img1
------------------------------------------------------------
1 snapshots
restore:
/root/restic2 --verbose -r rest:http://dc15s3:8000/repo2/ dump b727ac47 /img1 --password-file ./pass | dd of=/mnt/img1_out
compare:
md5sum /mnt/img1
d9a8853a50fedd618f3f361a913bac26 /mnt/img1
md5sum /mnt/img1_out
43f55c95aa1f3f951d62f917eedf2974 /mnt/img1_out
I use cat and stdin/stdout because i try to simulate backup and restore of block devices.
What backend/server/service did you use to store the repository?
i use rest-server in append only mode
Expected behavior
current tagged version WITHOUT compression works well:
backup:
cat /mnt/img1 | /root/restic -r rest:http://dc15s3:8000/repo1/ --verbose backup --stdin --stdin-filename img1 --password-file ./pass
open repository
repository 0376e109 opened successfully, password is correct
lock repository
load index files
read data from stdin
start scan on [/img1]
start backup on [/img1]
scan finished in 0.254s: 1 files, 0 B
Files: 1 new, 0 changed, 0 unmodified
Dirs: 0 new, 0 changed, 0 unmodified
Data Blobs: 92 new
Tree Blobs: 1 new
Added to the repo: 128.006 MiB
processed 1 files, 128.000 MiB in 0:01
snapshot 3a5bd550 saved
/root/restic -r rest:http://dc15s3:8000/repo1/ --password-file ./pass snapshots
repository 0376e109 opened successfully, password is correct
ID Time Host Tags Paths
------------------------------------------------------------
3a5bd550 2022-06-03 14:58:28 dc15 /img1
------------------------------------------------------------
1 snapshots
restore:
/root/restic --verbose -r rest:http://dc15s3:8000/repo1/ dump 3a5bd550 /img1 --password-file ./pass | dd of=/mnt/img1_out bs=1M
compare:
md5sum /mnt/img1
d9a8853a50fedd618f3f361a913bac26 /mnt/img1
md5sum /mnt/img1_out
d9a8853a50fedd618f3f361a913bac26 /mnt/img1_out
Actual behavior
content mismatch
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (15 by maintainers)
Commits related to this issue
- internal/repository: Fix LoadBlob + fuzz test When given a buf that is big enough for a compressed blob but not its decompressed contents, the copy at the end of LoadBlob would skip the last part of ... — committed to greatroar/restic by greatroar 2 years ago
- internal/repository: Fix LoadBlob + fuzz test When given a buf that is big enough for a compressed blob but not its decompressed contents, the copy at the end of LoadBlob would skip the last part of ... — committed to greatroar/restic by greatroar 2 years ago
- internal/repository: Fix LoadBlob + fuzz test When given a buf that is big enough for a compressed blob but not its decompressed contents, the copy at the end of LoadBlob would skip the last part of ... — committed to greatroar/restic by greatroar 2 years ago
- Merge work 0.14.0 to netapp (#2) * Add changelog entry for validating exclude patterns * Update 030_preparing_a_new_repo.rst * prune: Fix crash on empty snapshot * prune: Don't print stack t... — committed to NetApp-Polaris/restic by bmason42 a year ago
This is very exciting! Thanks so much for reporting the bug! 😃
Working on it.
WTF! I can reproduce it! How odd!
I’ve written a fuzz test that reproduces it and I can fix it by not reusing evicted buffers from the cache. I don’t understand it yet, though. It’s something in the interaction between caching and decompression.