restic: "Data blobs seem to be missing" error during prune on Backblaze

Output of `restic version`

restic 0.11.0 compiled with go1.15.3 on linux/amd64

How did you run restic exactly?

export B2_ACCOUNT_ID=...
export B2_ACCOUNT_KEY=...
export RESTIC_REPOSITORY=...
export RESTIC_PASSWORD_FILE=...
restic prune -vvv /path/to/dir

Output:

repository 06abe926 opened successfully, password is correct
counting files in repo
building new index for repo
[27:32:39] 100.00%  597407 / 597407 packs
incomplete pack file (will be removed): 003dec62564216e7819241c370154097d705b39c17e146ac5a6ba46f20913727
incomplete pack file (will be removed): 007e2a5d9070758730ed30e92763baf1633b58c32297a2d35f48798c6e24cfcb
incomplete pack file (will be removed): 03f8c59984daa1df72fa03b893ace7b9482eca8749166b39505d2ccd20672f7a
incomplete pack file (will be removed): 0582dcc70d68388def5c8898fed9066195f2975933eeae1ba7e8d93220385322
incomplete pack file (will be removed): 087dd11ad6890e87cf58cb075e0c2921a5204b04005d0c62f4b7c77579b42b2d
incomplete pack file (will be removed): 090f679d2d613f88bed226b7b689e1f7be675081f0a622ec962d312a949bd82d
incomplete pack file (will be removed): 095e265e665613d4c75317036861be77c43118dab78d0ea861788a1ecbb9e5ae
incomplete pack file (will be removed): 0a8fbcd5a0b7fcc2f31ecc3a2cad94857935b68bcea00be36b23d0f6230f2fa8
incomplete pack file (will be removed): 0b5e68125dfe12fa9035f78f70e98f0e2841853301486e4fd6a76ea56bb0f27a
incomplete pack file (will be removed): 0bd0515bc4a5fce93fb43f5999a7c57326198f201a088cb6cb4e899022b607a5
incomplete pack file (will be removed): 0cb0d8387b592be4a8c9af6dfbc50eab88631c56a892d3e9f53bd2d3ba8a2c7d
incomplete pack file (will be removed): 0ec5f045bc6e906da140a2bdaacfbdda09bcee8bf80ef8a47f7006db3fa4540b
incomplete pack file (will be removed): 10d03a20dec814751a2ca0bb211155842426788f93c09b00adf4ef686cc69ee7
incomplete pack file (will be removed): 14671c360e4e0070ac534257f04ec2d9f54eaa726dc02e33c49c794335666b0e
incomplete pack file (will be removed): 17559ab4f99881b2cb2b2e5e38776f15e2afdfac1c20cb873aa34d15b276fecd
incomplete pack file (will be removed): 17ae81809bc5fb38b2d3d797d176a929c10eaedf4d9569b128f767127c8836f8
incomplete pack file (will be removed): 1ad794f10507933f0fb922b26d2ac613a95aa3ed1e0aa2c32a99772fe9be436f
incomplete pack file (will be removed): 1c38cc7ddf296594b35841c902fe29b658bb5599e10fffa0343ef5747660a38b
incomplete pack file (will be removed): 1cae5e441b86dc564c6e5f4288e5d6550c989a7cd1fd7b7d05ee98dc62567d7e
incomplete pack file (will be removed): 1cd22f6b1ed9825eaf19ef14fd58f2741d0028bf94bd3939a3e3ed7e611f8c38
incomplete pack file (will be removed): 213184f2970f6af57db6d85bc84c2e8a7c33d713b0c3d4a496bea8138fe3bbe2
incomplete pack file (will be removed): 2245429bcc727256c243aa92d8de2d1415a2d412873e46f624793f39bfab2c14
incomplete pack file (will be removed): 255f6dbbcc5e9b60adbb42f06c144b10b304dc43e31eefc29cd3f974c100239e
incomplete pack file (will be removed): 2a34d963f6c706696335e1d4bda6741e01deeca162b8e983c7ecf2b542d73c94
incomplete pack file (will be removed): 2ac52911df5c641930ae8426c9728b32c0eb3c01a1cc34d66baf2722682a076b
incomplete pack file (will be removed): 2b80ed34c0db40cb2a6ec9634748e82a206f1b0620286dca9b2c08086033dff2
incomplete pack file (will be removed): 2d1368e95d4070be66524a10e9e58b9a9486a345cf22b9274ebeb934854b6d67
incomplete pack file (will be removed): 4e3d88deffff0698abf384172b9f54a6155b244d8d537e161558b1327b867ff2
incomplete pack file (will be removed): 4efd7ecf622374cef4a21b2162ef0838e6c3a14ccc773e17cf0291b1966ac745
incomplete pack file (will be removed): 957b9ef669e0652276221cba51f61794020e43fc6792b4066c03e4767510d152
incomplete pack file (will be removed): cf83988c27cd94128395478acde1b3575e0803a9897505a291b742c38bd43d71
incomplete pack file (will be removed): ef9a9e32d2841d0657acacefc859f1acaeb320011c600946632806be177ae32b
incomplete pack file (will be removed): fd96d91c45288709c0505656ca366df1fe7d68c119d6748cabd832abe05a9ff2
incomplete pack file (will be removed): fdb8a2c04b14b0e57cee91f594d7149a4d4122dbb9a0367afb37f4db31fa2ae5
repository contains 597373 packs (2150117 blobs) with 2.857 TiB
processed 2150117 blobs: 484 duplicate blobs, 707.676 MiB duplicate
load all snapshots
find data that is still in use for 25 snapshots
[0:44] 100.00%  25 / 25 snapshots
Fatal: [<data/3d56568f> <data/57181e75> <data/0e4ce059> <data/cc444c2f> <data/b2e379f9> <data/38f9a96b> <data/8132d9a4> <data/c573bcb9> <data/e4d8b94c> <data/3ec8733b> <data/901bcc03> <data/d9278c82> <data/dbcf2376> <data/d2a436a7> <data/33ba55e7> <data/2e3d7100> <data/8dd536f6> <data/f905ce2b> <data/1be0ac1a> <data/4bbe5a03> <data/c3200763> <data/1c0eafde> <data/d51d92b1> <data/4702c742> <data/013bac7a> <data/36258a6a> <data/511b80ed> <data/ec4f3f08> <data/3ef9832b> <data/d63c51e8> <data/dca1fee7> <data/5f082241> <data/d0463f6f> <data/dd32ba03> <data/76bb785d> <data/c746f890> <data/b4cb3746> <data/13703661> <data/d59b77d0> <data/5e226e55> <data/593b77a5> <data/58a9e264> <data/5ee6d6aa> <data/c3e0512e> <data/16017980> <data/2d5dfa95> <data/79d14056> <data/f70dadaa> <data/d9029400> <data/fe2558c8> <data/03fb9c4b> <data/0d077b94> <data/b82507b7> <data/6db121ae> <data/40a95358> <data/33a7b169> <data/947d9621> <data/43ed2189> <data/9cba2ff2> <data/f3cfdb69> <data/a4cbb0fe> <data/7a564ed3> <data/58518a68> <data/2becdeca> <data/abfe83ba> <data/e5645bc4> <data/0e7ee1ff> <data/4564d1e6> <data/fa4c4a9d> <data/9cfc7a10> <data/c6fccdad> <data/52195972> <data/bab427cb> <data/e4193f50> <data/ece7df04> <data/a57e94a7> <data/f9eb1c48> <data/c782cfaf> <data/94104234> <data/760366ae> <data/d38fa25c> <data/aa5caf50> <data/10db8281> <data/111031f5> <data/3f7e531c> <data/286aee98> <data/02d413a8> <data/e021d712> <data/09b591fe> <data/ccd60425> <data/c37d8a1f> <data/8e8626e7> <data/dd1758fd> <data/32a9c2e9> <data/3ba8e5a5> <data/fedadc33> <data/67be1c62> <data/97e0d088> <data/c5b65c04> <data/b2db64a3> <data/63b6f590> <data/3f7cde56> <data/158019e2> <data/cdc10a28> <data/76d7c237> <data/a8f44a72> <data/03251d90> <data/aa103e27> <data/e643d840> <data/aad7416f> <data/b94e06fe> <data/a6c9ec4f> <data/9d92d02a> <data/f56a50fb> <data/f2097e42> <data/edaff9e7> <data/fe9493bb> <data/991e99d2> <data/22e10578> <data/9d3a1480> <data/db8c2b66> <data/075bc94b> <data/651ebaf6> <data/b0a0cdc5> <data/2f21691e> <data/43b27c6b> <data/a49e26f1> <data/e3c57e8e>] not found in the new index
Data blobs seem to be missing, aborting prune to prevent further data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/choose

What backend/server/service did you use to store the repository?

Backblaze B2

Expected behavior

Prune to complete successfully without errors.

Actual behavior

Prune bails, with errors.

Steps to reproduce the behavior

Run the above command on my repository… No great, I know.

Do you have any idea what may have caused this?

None at all. I haven’t had any power losses during backup, and a complete restic check and restic rebuild-index come back squeaky clean.

Do you have an idea how to solve the issue?

No I don’t.

Did restic help you today? Did it make you happy in any way?

In general, I love restic. It’s my backup tool, both at home and at work.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 19 (5 by maintainers)

Most upvoted comments

@MurphySunset “nonce is invalid” means that the nonce field consists only of zero bytes. So yes that could be a problem at the B2 side.

restic check and restic forget --prune are running fine though.

--prune only has an effect if some snapshots were actually deleted. As check runs correctly, please try to remove the local cache of restic (see restic cache for the location), which might be corrupted.

If that doesn’t help try the following horrible construction to find out which pack file contains the problematic blob (based on https://github.com/restic/restic/issues/1999#issuecomment-424333904):

restic list index | while read idxid; do \
         restic cat index $idxid | jq '.packs | map(select(.blobs | map(select(.id | contains("0251d25ca5bebde32fdf9d947420ccbafb5529dc99cbd3a4f3a7545378c26ee2"))) | .[0] )) | map(.id)'; \
done

Then download the pack file using restic cat pack $id, restic cat pack --no-cache $id. Also try to locate the file using the web interface (it is stored in data/<first two bytes>/<full id>.

MichaelEischer on Oct 7, 2022