restic: restic prune - ciphertext verification faileds rewritten

Output of `restic version`

restic 0.9.0 compiled with go1.10.2 on windows/amd64

How did you run restic exactly?

C:\>restic prune -r U:\Backup\restic\C_Drive\
enter password for repository:
repository 8c029303 opened successfully, password is correct
counting files in repo
building new index for repo
[19:40] 100.00%  56898 / 56898 packs
incomplete pack file (will be removed): 004aaa58f6a419b83baab7c8b801cbd5df42a85b5c2ef44aa35ba3182b21c7f4
incomplete pack file (will be removed): 1fd4382824e77f7968317c22f0c2be030b8ef62b962de8cb5a8b74d62c79d85e
incomplete pack file (will be removed): 50c442a4feed1b49bccfd7f844121b1a69eda108ac9f221efd88da17f69c2d82
incomplete pack file (will be removed): 6ababb89b97ef1ce52d3d9710b8a4f51deacc582cd83767762d6f401a36fea21
incomplete pack file (will be removed): 8927c0c140ea69170831fb516dc4cab63557cbd0e63e55ee607967ce7c59ad64
incomplete pack file (will be removed): fb8c74093d9b1c1e4e7c5c6fc069018f3ca0c4dbf61aaeb6482a4fb7f7d006ef
repository contains 56892 packs (1498775 blobs) with 265.638 GiB
processed 1498775 blobs: 215082 duplicate blobs, 17.207 GiB duplicate
load all snapshots
find data that is still in use for 12 snapshots
[3:38] 100.00%  12 / 12 snapshots
found 1026855 of 1498775 data blobs still in use, removing 471920 blobs
will remove 6 invalid files
will delete 7735 packs and rewrite 10861 packs, this frees 65.865 GiB
ciphertext verification faileds rewritten
github.com/restic/restic/internal/crypto.init
        src/github.com/restic/restic/internal/crypto/crypto.go:30
github.com/restic/restic/internal/restic.init
        <autogenerated>:1
github.com/restic/restic/internal/archiver.init
        <autogenerated>:1
main.init
        <autogenerated>:1
runtime.main
        /usr/local/go/src/runtime/proc.go:186
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2361

What backend/server/service did you use to store the repository?

local HDD

Expected behavior

well, clean up disk space 😃

Actual behavior

see above!

Steps to reproduce the behavior

I would say, just do it?! 😉

Do you have any idea what may have caused this?

At least it doesn’t seem to be the “out-of-memory” thing

Do you have an idea how to solve the issue?

Did restic help you or made you happy in any way?

It feels good th have a so fast backup… thanks to whom-ever, I never needed a recovery so far… so until now I’m very happy with RESTIC 😮

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 29 (12 by maintainers)

Most upvoted comments

@nixzahlen You didn’t answer any of @fd0’s other questions, please do, he’s trying to help you.

rawtaz on Aug 2, 2018

memtest_fail winmemdiag_ok

I’ve run Windows mem check tools for hours, debugging a mem problem. It said ‘All fine!’ - I then ran MemTest86, which before I could blink, had spotted a bad mem block. I have screenshots attached, as it sounds incredible. However, removing the stick solved the problem. I also had crashes and corrupted files.

I’d highly recommend never to trust a ‘All OK!’ from the Win mem checker (a fail should be trusted), and also I’d highly recommend to use this, the free edition: https://www.memtest86.com/download.htm

nizedk on Aug 6, 2018

@nixzahlen - Can you reproduce the problem with a new repository?

If no, then it was probably “something that went wrong” - background radiotion/chaos effect. We’ll never know.

If you can reproduce it, I’d suggest generating 1 file, say 1 GB large, with random data, from, say /dev/urandom, and then checksum it. Or, it could be a .zip/.rar file, whatever, of some compressed music, so you have a file with builtin checksum, on Windows.

Then I´d copy it around with a script using dd on linux or robocopy /j on Windows, so that the destination file from the first run becomes the source file for the second run, and switch between buffered and unbuffered IO. Then repeat. And repeat.

(… or maybe compress one folder with the check-summed file, copy the archive, decompress, recompress, copy, repeat.) - just shuffle the files in a predictable manner that ends in the same state pr run.

That would exercise the cpu, the memory, and the drive, and most likely provoke whatever can be provoked. 😃

Abort at some point, or check pr run. Check file checksum/do archive test. If you can do that for a length of time, and the checksums from the original files are correct, then I’d start to believe that hardware isn’t the issue.

Can this be done in a much neater way? Absolutely. This is simply my Rube Goldberg attempt at it, superficially, replicating some of the same things that restic does, that could relate to failed ciphertext verification. If anyone has a neater util/app for it, please share.

nizedk on Sep 5, 2018

Thanks for pasting the output. I’d like to explain what restic tells you here:

Pack ID does not match, want acd94d2b, got 91e52123
Pack ID does not match, want 82774483, got e5e938e8
Pack ID does not match, want 5fdcc2d7, got 754299f1
pack b5ca83f6 contains 1 errors: [Blob ID does not match, want ba726d07, got 576d4855]

When restic writes files to the repo, the file name is set to the SHA-256 hash of the content. So when the content is modified, either on purpose or accidentally, restic can detect that when reading the data. Which is what happened here: restic recomputed the hash, and ended up with something different. So the data was modified, which should not happen. There’s not much restic can do here, except add error-correction information (which is a planned feature, see #804, but not implemented yet)

This means that it was modified either in transit, or on storage. We’ve seed both cases in the past, and in most of the cases it turned out later that either the RAM or the disc was faulty.

So, please consider running memtest and also a SMART test (if the storage device is a harddisc or SSD).

fd0 on Aug 2, 2018

Doubt it’s a HW problem… and here you go:

pack 58b1f3af: not referenced in any index pack e3a4a17f: not referenced in any index pack fcc2cb1a: not referenced in any index pack f1f5e8c7: not referenced in any index pack e76cadb5: not referenced in any index pack 2dee8570: not referenced in any index pack d1ce73ea: not referenced in any index pack 99e40e85: not referenced in any index pack 6067d4ce: not referenced in any index pack ffdbdee8: not referenced in any index pack 7336b425: not referenced in any index pack 7d57cb2f: not referenced in any index pack c3313922: not referenced in any index pack b54ca531: not referenced in any index pack 03429bf1: not referenced in any index pack ae53871f: not referenced in any index pack 01c181dc: not referenced in any index pack fd020252: not referenced in any index pack 5888fea4: not referenced in any index pack 46991019: not referenced in any index pack e1f54662: not referenced in any index pack 8582b75b: not referenced in any index pack 60bd7772: not referenced in any index 4632 additional files were found in the repo, which likely contain duplicate data. You can run restic prune to correct this. check snapshots, trees and blobs error for tree 83c36f77: tree 83c36f77: file “robocopy_E-Drive.log”: metadata size (610125432) and sum of blob sizes (610125674) do not match error for tree d883846c: tree d883846c: file “restic-temp-pack-714560053”: metadata size (2116370) and sum of blob sizes (2117016) do not match tree d883846c: file “restic-temp-pack-745083730”: metadata size (2204567) and sum of blob sizes (2204898) do not match error for tree ea565c3b: tree ea565c3b: file “restic-temp-pack-087244435”: metadata size (1873970) and sum of blob sizes (1874712) do not match error for tree 5a3b7617: tree 5a3b7617: file “restic-temp-pack-398758596”: metadata size (26298) and sum of blob sizes (26632) do not match read all data Pack ID does not match, want acd94d2b, got 91e52123 Pack ID does not match, want 82774483, got e5e938e8 Pack ID does not match, want 5fdcc2d7, got 754299f1 pack b5ca83f6 contains 1 errors: [Blob ID does not match, want ba726d07, got 576d4855] Pack ID does not match, want 9b306918, got 92527339 pack 05081022 contains 1 errors: [blob 0: ciphertext verification failed] pack 7cdfc311 contains 1 errors: [blob 1: ciphertext verification failed] pack 022588c9 contains 1 errors: [blob 22: ciphertext verification failed] Pack ID does not match, want ba84437e, got 71d1cac5 Pack ID does not match, want 0a949788, got 5c4e83c6 Pack ID does not match, want 79ad0777, got 759ef1e8 Pack ID does not match, want c66cd62a, got afe1824d Pack ID does not match, want b385efd6, got 2799ba3a Pack ID does not match, want 17b43b4d, got ef3b627b pack cd0e29be contains 1 errors: [Blob ID does not match, want dcbf8833, got 151db467] Pack ID does not match, want bfdf03f7, got 15509a2d pack 2ba2ab03 contains 1 errors: [blob 1: ciphertext verification failed] pack d9b36e75 contains 1 errors: [blob 1: ciphertext verification failed] pack d0e206de contains 1 errors: [blob 1: ciphertext verification failed] Pack ID does not match, want e3af12ec, got 4eb58b36 pack 2c6576b1 contains 1 errors: [blob 0: ciphertext verification failed] pack 0da55ff1 contains 1 errors: [blob 0: ciphertext verification failed] Pack ID does not match, want 7a284cca, got 4012806b pack 77cf03c9 contains 1 errors: [blob 0: ciphertext verification failed] pack bd32e160 contains 1 errors: [blob 25: ciphertext verification failed] pack f9b44f42 contains 1 errors: [blob 1: ciphertext verification failed] pack ae4f151f contains 1 errors: [blob 0: ciphertext verification failed] pack fb90f11d contains 1 errors: [blob 1: ciphertext verification failed] Pack ID does not match, want b44e2ff7, got 083fed95 pack 3d82a18a contains 1 errors: [blob 36: ciphertext verification failed] pack c66be056 contains 1 errors: [blob 0: ciphertext verification failed] Pack ID does not match, want cf3b5640, got 77b274a9 Pack ID does not match, want 1c9945b5, got bf00eca1 Pack ID does not match, want 7c919d0b, got ea22f341 Pack ID does not match, want 39dcdda7, got bfa863a3 Pack ID does not match, want 96ab5936, got 244856aa Pack ID does not match, want b83480f7, got a8726f8b Pack ID does not match, want fe507016, got c4ca987c Pack ID does not match, want ee1faa05, got ca49e556 Pack ID does not match, want 1fdc114c, got 9e10633b Pack ID does not match, want 2a133931, got 4078b195 pack da87bd93 contains 1 errors: [Blob ID does not match, want f429da62, got e53533e0] pack 4d5d263b contains 1 errors: [blob 4: ciphertext verification failed] pack b7b0cf4b contains 1 errors: [blob 0: ciphertext verification failed] pack 2adc696a contains 1 errors: [blob 36: ciphertext verification failed] pack b280a703 contains 1 errors: [blob 0: ciphertext verification failed] pack b55b7fdd contains 1 errors: [blob 1: ciphertext verification failed] Pack ID does not match, want 53aa4d46, got ccb27bfb Pack ID does not match, want 16d3b57c, got 0ab550a3 pack becf9b6b contains 1 errors: [Blob ID does not match, want 825d3415, got 4b311723] pack 0f8f529a contains 1 errors: [blob 4: ciphertext verification failed] pack e2bd90c1 contains 1 errors: [blob 1: ciphertext verification failed] Pack ID does not match, want 9ed36146, got 7efece9b Pack ID does not match, want 5024f978, got cc0569ca pack 0d2ba7d1 contains 1 errors: [blob 1: ciphertext verification failed] pack 57b5013c contains 1 errors: [blob 3: ciphertext verification failed] pack 72fc4a4b contains 1 errors: [blob 1: ciphertext verification failed] pack 5f7a9e0f contains 1 errors: [blob 3: ciphertext verification failed] Pack ID does not match, want 358687b8, got a26865c6 Pack ID does not match, want a5f3a24e, got ea54c08e Pack ID does not match, want 72873ff0, got d10884bf pack d0a7efd2 contains 1 errors: [blob 2: ciphertext verification failed] Pack ID does not match, want 4e68d344, got eecaca6e Pack ID does not match, want 66d8aaa2, got 7c93cc27 Pack ID does not match, want 5d8f7ee9, got fbdf0817 Pack ID does not match, want f3e70ecb, got 505198d7 pack e8063f65 contains 1 errors: [blob 3: ciphertext verification failed] [3:33:14] 100.00% 52765 / 52765 items duration: 3:33:14 Fatal: repository contains errors

nixzahlen on Aug 5, 2018