restic: restic prune fatal not found in the new index - "data blobs seem to be missing"

Output of restic version

restic 0.11.0 compiled with go1.15.3 on windows/amd64

How did you run restic exactly?

$env:GODEBUG="asyncpreemptoff=1"
restic -r "local:\\10.199.198.1\morgane" prune --verbose=2

What backend/server/service did you use to store the repository?

I’m using a samba server, mounted on Windows, both sides are connected via OpenVPN. The server is running alpine linux 5.4.72-0-virt x86_64 and samba-4.12.9-r0

Expected behavior

Prune should run without error.

Actual behavior

restic -r "local:\\10.199.198.1\morgane" prune --verbose=2
enter password for repository:
repository 3b1477ff opened successfully, password is correct
counting files in repo
building new index for repo
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 552.330144ms: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 1.080381816s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 1.31013006s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 1.582392691s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 2.340488664s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 4.506218855s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 3.221479586s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 5.608623477s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 7.649837917s: EOF
Load(<data/121711b4f0>, 591, 4696600) returned error, retrying after 15.394871241s: EOF
pack file cannot be listed 121711b4f0a2e5841f29c8abd3b74289b2daa0ee0843ea0058ce068fcd62d8ea: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 507.606314ms: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 985.229971ms: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 803.546856ms: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 1.486109007s: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 2.070709754s: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 3.67875363s: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 4.459624189s: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 6.775444383s: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 15.10932531s: EOF
Load(<data/3529c76718>, 591, 4930671) returned error, retrying after 13.811796615s: EOF
pack file cannot be listed 3529c7671832b0325c75eb18775fdbb327bb5d3b43638768aaf5177de0b21973: EOF
[6:41] 100.00%  66456 / 66456 packs
incomplete pack file (will be removed): 01bbfbba1443302f8be40e1f0b97efb588f51d643ffca059f8326b2d8b351b06
incomplete pack file (will be removed): 0b7d96dda6f1512a218b2200e49c185a97ac72789d99a426470593eb62b9abc7
incomplete pack file (will be removed): 0e50ff018f2a51731f38a0778c8b0cff6b2d1ea0ef62ba9fe10ed09303d923af
incomplete pack file (will be removed): 0e64f67054e9a2dc862a7e8fe4a24df0b20bd7a0d70cb9bb58aa012686371a49
incomplete pack file (will be removed): 101a65fe8f1c0e327697e2836c64959b63a3797b59ee16ce6de73824168d7003
incomplete pack file (will be removed): 14628541fa52185341de9f91eccab9ff33511db18377bde2107d4e74bafe8035
incomplete pack file (will be removed): 15658c11c965f10416a0d6a3408e5200ba08198fbbbbaa4bd86d74c2d39a4b48
incomplete pack file (will be removed): 169d22cf9c81a0642a71561f193fc66f328d7b27a4fa04eacbf553620526da38
incomplete pack file (will be removed): 16f52a1526195b2e2ad44d713987ff6ba8b605d4a69ff9a62ea1c20161800992
incomplete pack file (will be removed): 1715e869c90b86858263036aac4ff35d8fec04b5625eb9f30f9a2821c8e07279
incomplete pack file (will be removed): 198d8e28645a0241a40646024cfa21b976b4ab7112a6b2d3207dde9eea0309bd
incomplete pack file (will be removed): 1edb6109c1681d01682cea267d7150600da55b9c8fd5c312a593e5c35c1fcde0
incomplete pack file (will be removed): 200a29ff248600600b7bba18a6c76131bde0ad9d2eda1043837ea621c3102746
incomplete pack file (will be removed): 20bd82a37b3bf0601f2abd80cbe72755ae17bbcdb5262c49c5d56540f24957b0
incomplete pack file (will be removed): 228c92c841bba5367905c7cc74b66fb9f346a35970723d51e68e57eb68a40a7b
incomplete pack file (will be removed): 22a0297305875e66fa3a50135c795d373e3a31c286e5182a71a9f57c4d92825e
incomplete pack file (will be removed): 22b74566c39189854f0435717cdc2176109b3f3df5fe1417e5ad624384f1774a
incomplete pack file (will be removed): 233ab47dc516da28abe3fee29b0e5c4136127c3cef40492ae6d3fbe05b77ab66
incomplete pack file (will be removed): 24bfb647557d5412866b4cf47b30cb6b354ac7a9d3adc754232e451b86127294
incomplete pack file (will be removed): 2893a7916adc4f1833ea1c6527503c3d8871c40634dbaeef12c04526c03451fa
incomplete pack file (will be removed): 28ab16a8f8536283f4265fcf36bde89a0a5e68e4f8921e2e29577ad6af837778
incomplete pack file (will be removed): 2d55ce81a6feb743b3b17132488a3475ff0980310f6dc27f0351c3ea71ef6f71
incomplete pack file (will be removed): 308135349700d4e4c3fbe4470d595ae7ae44c7da2185809c8f0dd769aa5863e3
incomplete pack file (will be removed): 341366aedd8adf668292e20abdec12a7aff698be98f228459319f7a21cdb3adc
incomplete pack file (will be removed): 358c4291a879440a2e210a77740be264e870e77841476453b22c6aa08b15496e
incomplete pack file (will be removed): 35c52cdf3af540c792706e5b7714e90d035cc0acffb7849d337e4747fb4086e4
incomplete pack file (will be removed): 3892da70bfb6a5e5787f3c361ee6e7c8510e85d590205a3ac3e59f3cf8fdcc10
incomplete pack file (will be removed): 3b6f9f2869148c220807543742bc8c6368cde7f323b22844616b1bcf5bd4b901
incomplete pack file (will be removed): 3d55bd8514aff7b5030c62226968f39748d85e80856c87d3b0d33f141893dfaa
incomplete pack file (will be removed): 3fb1135e59efc97bf24a3332d9a45a061793b351ef164d93839beb4f4c61f20b
incomplete pack file (will be removed): 41feb5b4ce16cb9599a42c7af9f1bf211c617626242d5dfdb9ce349b84a54a5c
incomplete pack file (will be removed): 42d715b69de7884cb629bf35fc1d2495aa67e6304aea6a29351d25d1c761c2a9
incomplete pack file (will be removed): 442c013aa4f6f1df25ad0505fbbd401b570a5e9b69e8a88f02c565518762ed18
incomplete pack file (will be removed): 44f9ad5bcb2bbe0a80c14dc1b7de07287b9f80f9e0abb653467b5ecdff00b421
incomplete pack file (will be removed): 4def1a90b077e32baa7a4cdd59788c98cef2844603f69478e4620437eee55607
incomplete pack file (will be removed): 4eca069ae96088d7162cbe9d5e577e0ae68fca2fbe05018dd4def340222984c4
incomplete pack file (will be removed): 53b0df5e37ea81c0f5d671a401891900d88a4dd5d2436d5203d4847a5998489e
incomplete pack file (will be removed): 53bb37176490b33b3d58268cc3238d3a4858745483a088dbb64a3f23d838ddd5
incomplete pack file (will be removed): 54015a220e66b195cf31befc515acbc6710d8570e7b597236c7e3858881a2db2
incomplete pack file (will be removed): 568b63ab06e181089f31ef099b3aee0e468a5ed58f936ea4256c255a93596c31
incomplete pack file (will be removed): 576e1aee029e2eb814472bb822d7200d726292f46400e152410969b9458547f1
incomplete pack file (will be removed): 5a57130fa6823c795a84d96dea84d2c92a2733b89d9d0d29fedebabc767bd42a
incomplete pack file (will be removed): 5bfadd6fb5f0eb6e8d452e8a5e1ff98695fe425acf57785ebaf117fb5aaac449
incomplete pack file (will be removed): 5c533649be7580919fa4e99fa61ace922afa83c16cac6444f93a43355f9fa295
incomplete pack file (will be removed): 5efedf5cacbc20b122a1d8cb8b6dde57f359ecd375cd3d896d943a16a65f0805
incomplete pack file (will be removed): 620e7382e72099036d8e07432bd9785d4514835c3b9e440350e4d00054c9a18f
incomplete pack file (will be removed): 62665316765ff06f76be8b2a50db85c5e89639cb6427758626a90c6f74b9d462
incomplete pack file (will be removed): 65305bf215f05cf76a6b00f864876432704c9103a6bd32e21962b64fb89bebbe
incomplete pack file (will be removed): 65ca351a76379b57860686d61bb96a47dd4897c5f60b1a117f79434fe90742f9
incomplete pack file (will be removed): 66510076cfae80c796b4e8784b65834279a6622ea27f085506b540d3f60d9b21
incomplete pack file (will be removed): 669dd042681ccb7da400be408ec06871486d85eb2eb082a58b669d8fd238c0ce
incomplete pack file (will be removed): 69e5a10486eb1a6535d67d2c37b4bbbc5eb2210c732e1f48e67dd84a37828914
incomplete pack file (will be removed): 6c8c24149fefa1aae10cd9e4bb0dc39533f4521b6b45baca303c699e831c6ac5
incomplete pack file (will be removed): 6d1f3dda3816f34a2b67a9d825b46fd1c04070b880f8524de7e203fb7a6d130a
incomplete pack file (will be removed): 6e2daf44c62918d22f766e72645c61407cd6fbe97aef2f2d44be086739f65463
incomplete pack file (will be removed): 6e8ea2b0510fca9a087071000166e4610417971405423020a0132a99728a4b91
incomplete pack file (will be removed): 6f6914deff5b8c0715b6f9644094bd45074901614b534d7998b80ec3a65033bc
incomplete pack file (will be removed): 704b1afb874b8e12232d1535b7c47b9d92933c96381357e2428341576a26fb6e
incomplete pack file (will be removed): 72634c59746b14135b4eb03e94b0ba12bd95ccd1d396b9a9a3238f0bda38e72c
incomplete pack file (will be removed): 7388068f13e5e59194d02162ffa37b2b3478f24101a129936120df6bef10c1e8
incomplete pack file (will be removed): 77cceb241e29f38586c721ffa31dad38a296faabdc15cb20066adfde9c8effc6
incomplete pack file (will be removed): 7c25228a8447e2b2760f3c39402ab2b209810489c34c1d6f2e900e022a386d1f
incomplete pack file (will be removed): 7ee1f444b29913a2a15cb4933fb07bca3a493f8a795ee55616c14c1129123c90
incomplete pack file (will be removed): 7f01567a5337fda9eb6b27d8c1e9733953a1c737eb900e45e17a43aa62ea1b6c
incomplete pack file (will be removed): 815ee8678ebec56f365334c463246c99b4f86f8aa7d99812bcc0cf47ea909a3b
incomplete pack file (will be removed): 81674647a0a31ca7d58f701b8fc204ed69293986abdb196a5ecc5f4e6f65f2b9
incomplete pack file (will be removed): 858c8f85e7b2b9a4c975c35d64339e4b80474b99f32974e802eae1e88f89cb8e
incomplete pack file (will be removed): 8636d659576556503f6e9487374c96582b133dadeb900c8bb64c4345f9518162
incomplete pack file (will be removed): 876be24f95023caecb5762f64f79da602b28a724705eee39856b3f21733fda50
incomplete pack file (will be removed): 8771a8937e878750cbfa5d073267de9d5ede4af68d98d8fe9a545f93d0a17e52
incomplete pack file (will be removed): 87733621844537750fac6964ae7173f5db25304b8d20df99353f38fd85ae9243
incomplete pack file (will be removed): 87e910cc21ef7e6c8003c2e97348e594c975ef580e97d700a0da09628b561158
incomplete pack file (will be removed): 891fe6b57b1ebd955ac4d967a40edee2c2f699c4fbddff1d6201bae4fb92982e
incomplete pack file (will be removed): 8c45052bfb6a2a560b0967b43253e2df5ca94219395ec1150d78c8ade051037b
incomplete pack file (will be removed): 8c65f58507e1f10540025f81a754ec462415b1917779f3539e8b0c5184cc7eb8
incomplete pack file (will be removed): 8f6fb8fe394c22dfcf0b23a2dc7240f07d10e329b717d29f513de56be2ee3c69
incomplete pack file (will be removed): 9019537bde372702403093460e093fa06c5a9a6674c0bb78bf1a6ee26560bae8
incomplete pack file (will be removed): 9457e2625c7d735d63b6d1b780a0a2f57cae3cc9ac27203e0b2fb5102dd173e7
incomplete pack file (will be removed): 947f34d8870af1f63ad431b7147f08a7c1a8a67312e7d59093ab76fb006f2e56
incomplete pack file (will be removed): 986a1709af1da4a67e44d999088fa559a8f94de2b136200fe70ccf47d4063ada
incomplete pack file (will be removed): 98fb369c5dba7027c1c08e2081e4dcdb6aee97d06d3c703b0de97c02f9090d92
incomplete pack file (will be removed): 995fffb60671eb9beeb038d10b9fbb4366b8873dd2136aa082bb2fd2d4592af3
incomplete pack file (will be removed): 9c174485477026c0b8d14e91f85a3aa656103fab2f080603e11172d0f5782824
incomplete pack file (will be removed): 9c471e4efdea08e1af474ad8cb858ac4709b6f10b735f92a3c2a5cdfcece7654
incomplete pack file (will be removed): 9f3c8fa454fe998f2486fdc2a80a58dadf21f179536bd9e69fa587f98fcec703
incomplete pack file (will be removed): 9fbe6405c883aeb81b4e56f07badc244d1895f276db7614bd841a563739e5b9f
incomplete pack file (will be removed): 9fea7934f36b6fb5e1bb2fe297f0efcca0df5da9c97a43412fc9e29420764a7b
incomplete pack file (will be removed): a2b567082209eedd773d756523af1c20c4b4564ce23675483ad003611095056b
incomplete pack file (will be removed): a3a819ab453e11aee79870ea7541e4315b125808485248d79a609781127f2f8c
incomplete pack file (will be removed): a8173f14ec492e7a1eb5014589583ad4ded9fb089d5467e308b7c610e40ecab4
incomplete pack file (will be removed): a8c3e4570b47f0e06d1e5539a0b19ce9ef748dddf83858ffee11eacc55fde0b0
incomplete pack file (will be removed): a97258b623f11e148c34d326f2b4708e260af0c7e7ec21d189f8a9f076ab666f
incomplete pack file (will be removed): aa8f4c3624830863967ede9b9c7ca345b278090c2fb0d6c7915975ca7597af73
incomplete pack file (will be removed): af40b8349994da9a8e905606cc93b3fca206c2da3bb2250987f8534588fa1f50
incomplete pack file (will be removed): b2cf3f39179864dc76eb33d4876430de5250d922c6387ffcb30133ea580a6ec6
incomplete pack file (will be removed): b2d87756b66795d5b71c0fedea212c3bf706fadfa57b8358d970325452d99569
incomplete pack file (will be removed): b3ddb21b16f092079dab65836fe7341d0d5267f1304de016661c27c06d12cbc1
incomplete pack file (will be removed): b49ad20d191836cbed3a7909c42ca97dfbda08a949036a9a63ac031b3cac0e76
incomplete pack file (will be removed): b99195582fc7c5c27ac8c7f58607f8fdc8c449884c57473233332c31443e9d25
incomplete pack file (will be removed): bac39ca3de0a43a48336a2e8dedfcdaffe4c4180a91fc30518126368a5677394
incomplete pack file (will be removed): bda7cfcc76db93919e1d9da38ecd148746bc9d3510d66131ea2c4f01b4e27b2c
incomplete pack file (will be removed): bee5cc73bf7fc981306e0a5c98b68db1d561f80850b9ac36ecb560a7528be2d7
incomplete pack file (will be removed): bf8ad16e0ef72f4002b34d5504d7520402b405fa6c44b45afff262ae627a4b6e
incomplete pack file (will be removed): c07d63bbf4569fcc15f799fed1814e241c4c19eb55a744879aba56e8a0dc6750
incomplete pack file (will be removed): c339ba4fd959a16c7e16bbd27dd56b65e85b9a53c705376840dd6fd4fca7f4b3
incomplete pack file (will be removed): c45bd28d607546a2bb6f565d7cbc5006458e2336189826b17d106861c90e9958
incomplete pack file (will be removed): c5b69da7808eccde9e75176f94ab92f5de814d621a00e5513d6f0414dd571780
incomplete pack file (will be removed): c6b5b98de1a50da1a988f8e5b83c49e4a0be4c2f8f2e95f2ee399c64444e2620
incomplete pack file (will be removed): c82f92d1b2ad34cc12cab842c90139119270e2fe1861b2ddfb046862f8cb1f1b
incomplete pack file (will be removed): c9ca79c889332928b8ffc98a6331ccfe1ab5cfff161e21b2f54b74c55960c0c2
incomplete pack file (will be removed): cb35eb96cc8a8a6ce8cd02f3566d5ac07fe9bae3656a7a797b19b585a361bba1
incomplete pack file (will be removed): cd5788b5b2efd5e68d7fee4a8f12cfa5014ca4c66bde2cd046d635d855115cbe
incomplete pack file (will be removed): cda1740755069f782c31d9eba8937dd84806c45ec9b817b8b2321ce40a1b8589
incomplete pack file (will be removed): cf56eccc5ea9435ab8d3cbc9372ae07aef43a0e0ab2160bc009c3b0d48a24ea7
incomplete pack file (will be removed): d2553b2ac3c0f8a13c87c061f7463cc4f62c1078a0e4f466219d330ad2c67ab1
incomplete pack file (will be removed): d28324bd6261dda26a7f1f5cce514c89d85eec8a2ce87126c4521342565e3a03
incomplete pack file (will be removed): d57fb1cf8dd8bd2bdea91250eda07968c854fb62c993c7c4f8c959452791618c
incomplete pack file (will be removed): dead66fa9d77e18d150d6f0231f2c47f100e334d13fe1c9d784719ea72f72f99
incomplete pack file (will be removed): e0592f55b5b56795bbcb734d5db843f2409ef0a6b32227318e54b04e8ef773f5
incomplete pack file (will be removed): e0bc829f0bc713ff6350a4aeb53d6845a527737a7aa4b6a4f5ffbe00798ad249
incomplete pack file (will be removed): e736568cb38289191c38f429fc02cfe105292d772668624497750f2215832268
incomplete pack file (will be removed): e765656b8afbcc0a572766fd2d18a5485e1595a60ddc62876d2a1c0d977b0785
incomplete pack file (will be removed): e782bb53446a68e1ee6f1a66b06a63b4afac4e2a0dd528b24409c727e3b84920
incomplete pack file (will be removed): e7bd3365762efb050e035e5668a3f8b6615653f518d7a99f4fd11a34d2b018fa
incomplete pack file (will be removed): ea6fbf4042f2dd78b834b385bfa31b163a4b908f400b4425399ea3580e99d0b6
incomplete pack file (will be removed): ebdcf467f61e540466673b4f01305a0b0e680dde1396ea044aca183a0705eb38
incomplete pack file (will be removed): efa963cd903ae9010575697bd91e484f7b848687610b302fbf83f4b53dd08a90
incomplete pack file (will be removed): f4fac8c27373d1e54cbde9fafbe014fc52dbfea23cc2fe8ff9d2a7197a91c0e4
incomplete pack file (will be removed): f7c10eb112fb7e985f8dd83ecbd18e2b45367de5e29d9ceaee0a209c104d29d4
incomplete pack file (will be removed): f8a4cda36fd70a3b0c73f4ed315c36a0e1824c219acb4a4ef3469b1c3bcf34fb
incomplete pack file (will be removed): f99702e6b571a4a494eeb08e5c8dd4387f70bc4fa3bfd9cedf6f05213d103e0a
incomplete pack file (will be removed): fa0e240ef605afc9dfa387a9a0c00c51d112f7abbe97e206eb9b65c1bfa976dc
repository contains 66322 packs (297477 blobs) with 317.253 GiB
processed 297477 blobs: 30854 duplicate blobs, 33.827 GiB duplicate
load all snapshots
find data that is still in use for 6 snapshots
[0:02] 100.00%  6 / 6 snapshots
Fatal: [<data/6ebb2b82> <data/da498070> <data/1daf87c0> <data/d5b7988a> <data/f9f6bec8> <data/8b044261> <data/4869b5b8> <data/c127f59c> <data/f8579293> <data/3d899bd7> <data/f63aef02> <data/b56c4b0c> <data/8aac9d21> <data/0919912a> <data/829ca4a3>] not found in the new index
Data blobs seem to be missing, aborting prune to prevent further data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/choose

Steps to reproduce the behavior

1. restic -r $remote-place backup --verbose=2 --tag $tag $local-place
2. restic -r $remote-place check --read-data --verbose=2
3. restic -r $remote-place prune --verbose=2

Do you have any idea what may have caused this?

I believe this was due to backing up the directory via SFTP first, which had some errors (it seemed like it was related to ServerAliveInterval, however I tried different things after searching online, but it did not fix it, SFTP always failed after a while). This means I’m pretty sure the server had some bad files (in the middle of writing but got cut off).

So I had to switch to samba + mounting it via OpenVPN instead. I re-ran the backup from start, with the same tag. Then ran it again with backup --force --no-cache. After that, I ran check, then check --read-data, then prune and got the error above.

Do you have an idea how to solve the issue?

I’m not sure. I’d like an option to do a backup that goes through all files again to make sure no data is corrupted. Is there any way to do this?

Did restic help you today? Did it make you happy in any way?

I like restic. I tried backing up with: rdiff-backup, borg, and duplicity; restic was not only the fastest one but the one that did not choke on backing up in my particular case (except for that sftp error which I still have not figured out why).