restic: restic prune failing - tree XXX not found in repository

Output of `restic version`

restic 0.9.6 compiled with go1.12.12 on linux/amd64

How did you run restic exactly?

$ sudo -E restic -r /media/auto/nas_robertfoss/restic prune
enter password for repository: 
repository 237fdd10 opened successfully, password is correct
counting files in repo
building new index for repo
[3:52:27] 100.00%  440951 / 440951 packs
incomplete pack file (will be removed): 01e37cc714147b7ec051e8eb4cf045e71c69b1b10e18eb83ed9fe9338c2d24d6
incomplete pack file (will be removed): 02166c70e25a51cb5b5b44d3b9702c0a540f6ca842c19e4f78bd1774ca782957
incomplete pack file (will be removed): 02b80c6de7f81142a5e096891069209b88a4da91e1c84ae316a40f2b5238bcd4
incomplete pack file (will be removed): 0be78f6a34de32f800ad05cd49347b27ca16ccc1185e7858481faf785aa9992a
incomplete pack file (will be removed): 0d3bddd90c2748e6f1acd70ed286283d346f17d8c012c2ed606dd1aa6bac2f06
incomplete pack file (will be removed): 11a70cf10e5fc08fe6875b8c988c17af3447c99c6082f82006e9dd14a26b635e
incomplete pack file (will be removed): 11c8c8ef5e80cd654115769b87a3bf7c1d4fefaf5a4ec226209f3627b7159969
incomplete pack file (will be removed): 1efc58add323f3e1e2aae1d751d1be6f26b90d1c5a408cee3edffe1de02e4b4b
incomplete pack file (will be removed): 2d1763064c0a39abf21377bcdf056ad6f4e12a7b93cd28b3dbb98002dd91813d
incomplete pack file (will be removed): 31c299e7f80a3836a394da0795e8ed65d7277dfed9e3863db013bc0b4d99d9c7
incomplete pack file (will be removed): 3557c70b0a76e3303ad51ac755a2ab577d6c041640cd015da345308f492b563b
incomplete pack file (will be removed): 36745b2ec4fffe49ecf6a9483d74a7a7c352e077ec7b830d4d162b0d16739a6c
incomplete pack file (will be removed): 44090b5084dbe0746a68151a5980d5078666b8cd92b889032a99a4297ceca7c3
incomplete pack file (will be removed): 4433545485727ef6406307e1d18ded514ea59e6230fba5deeba28f82436cbf71
incomplete pack file (will be removed): 4b0be68275743cdf4e69dd795840050d038eb69d3f6d0f3c36cfc5d7d01d991b
incomplete pack file (will be removed): 4c4dd542a0300096250e01b86f71dd2f5a2e9e7e4b2cb9242df953ab432814cf
incomplete pack file (will be removed): 4df36fec011b6add8541e9c20064322b2292297f931fe525f46c88c69db8d752
incomplete pack file (will be removed): 4f654a34453ba4ff32aabd25013081c660ee9895cfe1482d764fa53d31f392e3
incomplete pack file (will be removed): 4fa5793656a1514e2846a4ff57aa2357333f92eb29617682760d42b72771cd8b
incomplete pack file (will be removed): 512ba62d659a52d26ce0db2e4bf09da509cac1f418fe9db151bf2a5ac830cd5d
incomplete pack file (will be removed): 59a07eafe7a187400ab744521edd9800dede1064260dad1b3f32080d090c2b5a
incomplete pack file (will be removed): 60719e8a0247033dcd8bc1c89dfd34cdf67499e423145ab568b32ac9b209e00d
incomplete pack file (will be removed): 625243d5fee5a4bdd8dbb33d9864a375b424a15c9fd7d190d244f2f1eb4e8ec3
incomplete pack file (will be removed): 64ecbbf7d6365c73a1e431b64697eed338fc8e130800dc261f268611b07b8447
incomplete pack file (will be removed): 684040807b3515b1336cc22d5ebe750e3dd0fa7a12e99d5808b1ee834a784f81
incomplete pack file (will be removed): 6a4e046d6de8f89f4e9a7a87dba5bcbf43e1b150148646a20f67130e9421ade4
incomplete pack file (will be removed): 6b6d7766ac57c254e8ed539fcb1f74d2722d024fc231abe7e5b8d26e3367ef01
incomplete pack file (will be removed): 6e05c04f66e9e8335dd4b3f5217c0edf389ce3f2cd53362273edddc7b2026c76
incomplete pack file (will be removed): 6e3407887327ac22b1271d22096b0fa817701166da8a0d7b1200b6795fd0ca3d
incomplete pack file (will be removed): 70c4133e9a9262d537e1c46eb6b09df5c9c04b8028108f3ca7d1a4b9e614f293
incomplete pack file (will be removed): 70ce970d142d08dd86bf4f4e9e1f7c2b6d5dca3655e72bea9a74c56de0608d69
incomplete pack file (will be removed): 74dac8ca2412961d27289c47562af19b62251ba83db41655a809a2aa3e04d1c4
incomplete pack file (will be removed): 780de5f377e4db9d7b5d383b530bd8e87768268f4e36d8182e9b8dcfd62f1292
incomplete pack file (will be removed): 784ace35191fc681b3a90fba99654d6b3983dc9e224f5ee96ca50101e1cc3c57
incomplete pack file (will be removed): 7e6fd087f8bcb9d77d97284c05568f0bb79db6425129a4b6796b27c115196629
incomplete pack file (will be removed): 7fe7e27c4aa423d015e85a9eb8abe7939af51ef24a0fb0c39c46d7d1f4260b8a
incomplete pack file (will be removed): 84cac8089d827106faf06a5c8bc9446573c75f0ac233749130c81b7e3538b0b6
incomplete pack file (will be removed): 899ff4b1a0d24b818b3b106d950804a747ba2a7f72676b7bd9b81c48306eaadd
incomplete pack file (will be removed): 9397dbad057de654ed96524e16db2a7101811352ebcb08d153ab96a2628720fe
incomplete pack file (will be removed): 946ad13a3cec6953448fc8947bd3f418ea1be1a9d38ac39b80dd108ad7a797f9
incomplete pack file (will be removed): 9ae6b1f02d3e4f2008bf9411213b51e697a703bfddfb5383165b1b17fceaf3bc
incomplete pack file (will be removed): 9b7662aeab336fef8dfba60399631c92069fd8514aea97d1175d92f7c8aeade8
incomplete pack file (will be removed): a0b52520a5c2b9f4f477e8c4155e269a24aaad5ac1c12c36fbf56072b28f2036
incomplete pack file (will be removed): a15b2d9db7b0955b1a0ebf4f76364b4a87ed0c830adf539be8aa592922e19ba7
incomplete pack file (will be removed): a6f12ba5da6420a4171b866a5c114ab7e558820331dfa82c5fbc909371a1c96c
incomplete pack file (will be removed): a889ced3aaba4a3b6c7536a4262b703c90244d842e0808f942819443fdd8049d
incomplete pack file (will be removed): b574b68758b3ad8f3eaf16f26c1af4e127b1a93af17e6f9b1ec5a17ad322452c
incomplete pack file (will be removed): bc36699cfcc6babb8670c6a27eef2ed23acc7f091e82bbc7102017ffab792dc6
incomplete pack file (will be removed): bd7b650d802217450570976e56e43185b36910b0a12ff5b73bf04a07a95c3c50
incomplete pack file (will be removed): bf384616635115743e79b6c25ebd0e65bc7c8b3fa27090e3719837811b753f9b
incomplete pack file (will be removed): c55c9b36a53c3953306fd4f003750b9c164c3c95d634becdd68b36a9343c45d7
incomplete pack file (will be removed): c5d74ba19f733815d2f26cf21a23cde22328d20d162bf16f8a9ff3cfa3f7a1d4
incomplete pack file (will be removed): c6d47d52ac7cb2f67c59226da1f488bcd77cb4e539be0069c9131585b2da8d64
incomplete pack file (will be removed): ce6f710e03a54ce10f747531efcdf3d6e54edbab42de62ddfb8eb955b3b0caaf
incomplete pack file (will be removed): d44fa38cf9ffb990049db2bbf233e7dee56678304326b834bd67259f71989112
incomplete pack file (will be removed): d8832cf344b82f26c10a68c65694ebd1a21e51ac117d874cdc82c84f1eb1dfd2
incomplete pack file (will be removed): dbe65af86c8eea1ec694c25180e9a2ca46f0047d7c65f4b48789eb48cafd0934
incomplete pack file (will be removed): dc6adc442ffd91c8841ef40133b45d0e27ebe2f45eda0a5ca5b698daf828261d
incomplete pack file (will be removed): de6d33643239da4809e9ff6ccdbb66f49ca31df8ea33e8378ba60fee00c751f7
incomplete pack file (will be removed): e4015a03a9b69818f2ba631f37d0cc713a3a4c8e73c1aba92327cf0bbead69b9
incomplete pack file (will be removed): ea3bf0acf2326a2c730e01fc233916686e4c1243ea327110bce1b0639b78238a
incomplete pack file (will be removed): ea896919de15181889c941a093021f90ef5b8340bc9cd894b9a91ca93b75751f
incomplete pack file (will be removed): ee0af7c8d44a05b74083bf20d9116ddb80f93f3f907da8a7132c6d5861df03b2
incomplete pack file (will be removed): ef6c72c5c9e34bb96456e53e8f646376416a6194db2260200c6764a1e9626a9f
incomplete pack file (will be removed): ef6e0530aec863e408e09860ff91121df08f3870d0a251c78b2b2cbff05a6f9b
incomplete pack file (will be removed): f003cd6ebfa976771f8d7e0a76eb41d14c92d62d962c3c74fcc67dc87769d209
incomplete pack file (will be removed): f095ad5213e7b4bae17fb627db806f41f644f8412f9ae89b8dddb1e2c57cc69c
incomplete pack file (will be removed): f460b877db2fe4148cb67826c70f757dedbd0f4be44da59536e17ca2863f82fe
incomplete pack file (will be removed): fcfc1622fe2967de167c5a6b2f1838117daf1b26dc3d11a9731d16ef04cb5188
repository contains 440882 packs (21714968 blobs) with 2.009 TiB
processed 21714968 blobs: 1546276 duplicate blobs, 53.897 GiB duplicate
load all snapshots
find data that is still in use for 42 snapshots
tree 51a481d9674fe804a2900e71f9a95ea8c1083d2167629db4549e0c9693ecf79b not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
	github.com/restic/restic/internal/repository/repository.go:713
github.com/restic/restic/internal/restic.FindUsedBlobs
	github.com/restic/restic/internal/restic/find.go:11
main.pruneRepository
	github.com/restic/restic/cmd/restic/cmd_prune.go:191
main.runPrune
	github.com/restic/restic/cmd/restic/cmd_prune.go:85
main.glob..func18
	github.com/restic/restic/cmd/restic/cmd_prune.go:25
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra/command.go:850
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra/command.go:800
main.main
	github.com/restic/restic/cmd/restic/main.go:86
runtime.main
	/usr/lib/go-1.12/src/runtime/proc.go:200
runtime.goexit
	/usr/lib/go-1.12/src/runtime/asm_amd64.s:1337

What backend/server/service did you use to store the repository?

Local filesystem (backed by CIFS)

Expected behavior

Prune not failing

Actual behavior

Prune failing

Steps to reproduce the behavior

restic -r /media/auto/nas_robertfoss/restic prune

Do you have any idea what may have caused this?

Likely some corruption in the repository. Not that I have any particular reasons to think it is corrupt.

Do you have an idea how to solve the issue?

Did restic help you today? Did it make you happy in any way?

Not today, but most other days 😃

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (10 by maintainers)

Most upvoted comments

hash does not match id: want e1ee8d8a29fd16633ae622e2c18c993ae3984d77d93d5641917e6eb0cf7a6687, got 97b186d7fdc6a7cf237f803ce233702fd90d6ea6e672d556c07190dda8bbafa2

This error message tells me two things: Firstly the affected pack file was fully uploaded to the storage backend (as prune would otherwise have failed during the initial reindexing) and secondly the file was somehow damaged.

Can you download the file from data/e1/e1ee8d8a29fd16633ae622e2c18c993ae3984d77d93d5641917e6eb0cf7a6687 manually (restic cat pack e1ee8d8a29fd16633ae622e2c18c993ae3984d77d93d5641917e6eb0cf7a6687 should also work) and run shasum -a256 <file> on it? If this still yields 97b186d... then the file was probably damaged before/during the upload. As your logfiles indicate that the backend connections runs over HTTPS, that would leave two locations for where the bitflip (?) could have happened: On your local computer or somewhere in the Swift Proxy/Storage nodes in the backend.

The “hash does not match id” error was already discussed in great depth in #1999, and also in #1596 (and probably many more). There are also a few ideas to salvage blobs from damaged packs, e.g. #1727. I’m not aware right now of an issue to track adding end-to-end checksums for the backends (see https://github.com/restic/restic/issues/804#issuecomment-615380222 for some elaboration on that topic). For the download step there’s #2302 with a suggestion to improve handling of bit-flips.

[Edit]I’ve noticed that my interpretation of the error message is only fully accurate for the latest master branch. With restic 0.9.6 there is a chance that the initial rebuild-index check does not fail. However, if I understood you correctly then the check command did complete successfully after rebuilding the index? In that case we know for sure that all relevant pack files have a valid pack header (i.e. the were fully uploaded) and then my previous reasoning applies unchanged.[/Edit]

MichaelEischer on Aug 27, 2020