After a multi day exploration using a number of our engineers, we believe we determined the root cause of this issue.

When data is uploaded to our infrastructure, each file is erasure coded across 20 storage servers, which we call a Backblaze Vault. The files identified by @kmwoley and other Restic customers were all stored in vault-1129.

This vault happened to have disk errors on a number of hard drives. Therefore, it was possible for a download to intermittently be corrupt, as reported. However, because of the way files are stored across multiple servers, the file is still correct and available. No data has been lost.

Backblaze has an integrity checker which slowly runs over the disks looking for issues. In fact, by the time engineering started looking into the problem last week, some of the corrupt parts where already detected and moved aside and queued to be rebuilt. This led to taking longer to understand the problem. Going forward, our tools to determine the state of files will be improved. Also, there is a project that will be going live in the next month or so to better detect this issue while reading a file for download.

At this point - we believe the issue has been resolved; vault-1129 is healthy again. If this is not the case, please do let us know by opening a support ticket and referencing this post.

It is a mystery why only restic customers reported this issue to Backblaze and further why the data was stored on vault-1129. (For context, each vault has 1,200 drives and as of Dec. 31, 2020, we have 165,530 drives in production, or ~137 vaults. This was probably just a coincidence.

@kmwoley is correct. We deployed a fix that should finally resolve this issue. Thanks to @kmwoley and everyone who reported the issue. If you continue to see a problem, please file a ticket and reference this issue.

Thanks to everyone who helped diagnose this issue and your patience.

Welp. Bad news. I’m still seeing corruption when I run restic check --read-data… got an error within 10 min of running --read-data.

I’ve sent details to Backblaze support to continue the investigation.

Thanks to @kmwoley for reproducing this issue without restic. Our engineering team has the detail he provided and we are currently investigating.

Hey folks - I can confirm that Backblaze deployed a fix on Feb 26, 2021 (around or before 12:29 PST) that resolved the data reading errors I was experiencing. Many thanks to all of you who helped diagnose the issue and get it in front of Backblaze for attention.

@kmwoley

Is it expected that the check does not show progress for a long period of time (i.e. hours) at this stage?

# restic check --read-data | tee 2021.02.20.log
using temporary cache in /tmp/restic-check-cache-740384393
created new cache in /tmp/restic-check-cache-740384393
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
read all data
[1:51] 100.00%  215 / 215 snapshots

I can see from my system monitor that it’s downloading at full-speed and using a core of CPU, and has been for about an hour or more without reporting any progress.

As you pipe the output of check into another command, that is considered as non-interactive usage. And for that case restic doesn’t output any live progress. With restic 0.12.0 it’s now possible to configure that using the RESTIC_PROGRESS_FPS environment variable or by sending restic a SIGUSR1 signal.

Summary

I believe this to be an issue independent of Restic. I have downloaded one pack directly from B2’s web UI that resulted in a checksum mismatch. I have run 2 full data checks from different machines which appear to give random checksum results. Every single manual pack checksum check I have done passes, except for the one documented below.

Manual Checksum Failure - 49791035

In the full --read-data checks, there were 5 packs that showed in error across the two checks. I manually verified each of those packs by downloading the pack from B2’s Web UI. One resulted in a corrupt file for the first time I downloaded the file (on Windows, Chrome). On second download, the file was correct.

Here’s the SHA-1 results (to compare against the Backblaze UI):

Binary Diff

I was curious if his was a single bit flip, or something else. So I put the corrupt file and the correct file into a binary comparison tool. The result was that about 12% of the files differed… in 5 different locations… not just the start or end.

I also downloaded the file through the UI 5 more times… each of those subsequent downloads through the UI was correct.

Checking the single file in a loop

I’m currently running the download / shasum of the single file in a loop - I’ll report back on what I find there.

Summary of --read-data testing

• I have run two full data checks from two different machines using Restic 0.11. The detailed results are posted below.
• The rate of errors is different between the two machines. The Linux machine produces ~ 0.84 errors/hour of running check. The Windows machine produced ~0.56 errors/hour.
• Of the 67 packs in error total (out of 148050), only 5 of the same packs showed up as in error from both machines.

Full Check Results

Windows Machine

Windows 10, Restic 0.11, Wireless Network

PS C:\restic> .\restic check --read-data
using temporary cache in C:\Users\Kevin\AppData\Local\Temp\restic-check-cache-954866003
repository e07618ce opened successfully, password is correct
created new cache in C:\Users\Kevin\AppData\Local\Temp\restic-check-cache-954866003
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
read all data
Pack ID does not match, want 8ddad374, got b4237d17
Pack ID does not match, want f5ef3b30, got 957945b4
Pack ID does not match, want 76512646, got d983233e
Pack ID does not match, want eefb18c5, got 28e6b164
Pack ID does not match, want 202597c6, got 6976c331
Pack ID does not match, want a0047d1a, got 3135cc61
Pack ID does not match, want f72d22f6, got bd0050a0
Pack ID does not match, want f37288d6, got 1caab1ba
Pack ID does not match, want 581b0af8, got 05113ec4
Pack ID does not match, want 8e21bea4, got f6ace65a
Pack ID does not match, want 27b07e9e, got 98c3ded8
Pack ID does not match, want 81870b82, got cdee9d5d
Pack ID does not match, want eb228e74, got 736f0e8f
Pack ID does not match, want af8d336b, got 3e076569
Pack ID does not match, want 99fe9e94, got 70e33c03
Pack ID does not match, want 02080be7, got b2514977
Pack ID does not match, want 4ac9361f, got 5fde1c78
Pack ID does not match, want 7cd68700, got ea422a27
Pack ID does not match, want ba75213e, got be695407
Pack ID does not match, want 49791035, got a1baa971
Pack ID does not match, want ca8e84cc, got a0cee332
Pack ID does not match, want f900df97, got 2960de63
Pack ID does not match, want 6db65d0a, got 2c4792af
Pack ID does not match, want 353ae954, got 87130773
Pack ID does not match, want 1fb4983c, got 4a49100a
Pack ID does not match, want ce95f9d8, got 4ec61790
Pack ID does not match, want 31b6ede2, got c7f7b114
[48:20:01] 100.00%  148050 / 148050 packs
Fatal: repository contains errors
PS C:\restic>

Linux Machine

Unraid Linux (backup host). Restic 0.11. Wired network.

using temporary cache in /tmp/restic-check-cache-339533226
repository e07618ce opened successfully, password is correct
created new cache in /tmp/restic-check-cache-339533226
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
read all data
Pack ID does not match, want c29bc1a4, got b9d5d04b
Pack ID does not match, want bd505916, got 9861462a
Pack ID does not match, want 176ccc63, got 8a69d9fd
Pack ID does not match, want 287d3c84, got 7c6bee66
Pack ID does not match, want 4ac9361f, got 4c2b16cd
Pack ID does not match, want 316d4bf5, got 5ede609f
Pack ID does not match, want df72f5a1, got 6aa414b7
Pack ID does not match, want ad74edf2, got 03632064
Pack ID does not match, want ba2b9894, got 63c0f8cb
Pack ID does not match, want f4f93c0e, got 09f69c2f
Pack ID does not match, want 082c2dcd, got bf4f50d7
Pack ID does not match, want aeadd833, got b21a1f58
Pack ID does not match, want 4199ff0a, got 574624b6
Pack ID does not match, want f5ef3b30, got dff3c6a0
Pack ID does not match, want 35f93620, got f18216a5
Pack ID does not match, want 27b07e9e, got 735d923f
Pack ID does not match, want 1dfbc37c, got d32bdfe4
Pack ID does not match, want 429b7282, got 54890386
Pack ID does not match, want 7cd68700, got cbf88637
Pack ID does not match, want 7ef7ad06, got fde1949d
Pack ID does not match, want e15cdbeb, got 16621489
Pack ID does not match, want 119cd505, got de561fff
Pack ID does not match, want 6f0f9d33, got 537b8924
Pack ID does not match, want 864f3137, got 37590313
Pack ID does not match, want 97bccb43, got cf730317
Pack ID does not match, want aad302e6, got 29f03512
Pack ID does not match, want aab71342, got 24eec66d
Pack ID does not match, want 47336b30, got 8f871085
Pack ID does not match, want 5adc97ac, got 6c59e41a
Pack ID does not match, want 49791035, got 36db95e9
Pack ID does not match, want 201bccd6, got bd0c9dd6
Pack ID does not match, want aa03c2b8, got 4a52996d
Pack ID does not match, want 79f599e4, got 1d5c3fa6
Pack ID does not match, want a463af1a, got b9cc48da
Pack ID does not match, want 83cf9760, got bfc67dca
Pack ID does not match, want 66a803f6, got fb373c39
Pack ID does not match, want c51332bd, got 3be73f0b
Pack ID does not match, want 313d8b37, got 6b2d8184
Pack ID does not match, want 2ebd7b8f, got c40c820f
Pack ID does not match, want eb9c1d4a, got 15322491
[47:54:31] 100.00%  148050 / 148050 packs
Fatal: repository contains errors

Packs with errors from both runs

Want	Got	Env	Size (KB)
49791035	a1baa971	Windows	5,182
49791035	36db95e9	Linux	5,182
27b07e9e	98c3ded8	Windows	4,859
27b07e9e	735d923f	Linux	4,859
4ac9361f	5fde1c78	Windows	4,711
4ac9361f	4c2b16cd	Linux	4,711
7cd68700	ea422a27	Windows	4,215
7cd68700	cbf88637	Linux	4,215
f5ef3b30	957945b4	Windows	4,300
f5ef3b30	dff3c6a0	Linux	4,300

I downloaded each of these packs from B2 using the GUI to validate their checksums. All of them passed, except 49791035 as noted above.

Other notes

• I recently had upgraded my router firmware. I downgraded it to the previous version and ran a --read-data check to see if it made a difference. It did not - I still got errors. 😕