foundationdb: 'Corrupted backup data' in fdbrestore

Hello!

I use fdb built from master dfca94d00f0341905ee302ec9308ae580b53d325 (2020-07-11 06:48:09)

I have a three node fdb cluster. I’m trying to make a clone of this cluster to a new one.

I performed fdbbackup to a some file location.

fdbbackup start -C fdb-primary.cluster -d file:///mnt/synology/nfs/backup/fdb/hot/9 -w

After the backup completed I started restoring it to the new cluster

fdbrestore start --dest_cluster_file fdb-clone.cluster -r file:///mnt/synology/nfs/backup/fdb/hot/9/backup-2020-07-14-20-20-58.753553

After some time restoring stopped progressing and status shown ‘Corrupted backup data’

[oleg@osamarin3 devops]$ fdbrestore status --dest_cluster_file fdb-clone.cluster -r file:///mnt/synology/nfs/backup/fdb/hot/9/backup-2020-07-14-20-20-58.753553
Tag: default  UID: 28ae6c19bbe6b5444089e3e75b7803b0  State: running  Blocks: 21899/21900  BlocksInProgress: 1  Files: 3848  BytesWritten: 20426795788  ApplyVersionLag: 0  LastError: ''Corrupted backup data' on 'restore_range_data'' 32s ago.
  URL: file:///mnt/synology/nfs/backup/fdb/hot/9/backup-2020-07-14-20-20-58.753553  Range: ''-'\xff'  AddPrefix: ''  RemovePrefix: ''  Version: 1480966563538

I’ve tried to make a new backup and restore it. But each time fdbrestore processes 21899/21900 and can not finish because one block is corrupted.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 47 (24 by maintainers)

Most upvoted comments

I’ve analyzed the bad block in the first backup you shared. It appears that the keys and values have common headers, so using those to make assumptions about where strings should begin it appears that at the very least 2 of the string lengths are incorrect. Fixing those two lengths allows the backup file to be decoded successfully, however I can’t be certain that there is not corruption in the key or value strings themselves.

Do you have a way of verifying that all of the keys match the original database (if you have it in its original state) or at least that all of the keys and values are well-formed?

Assuming you have perl, this very ugly command will patch the backup so that the bad block is parseable so you can complete the restore.

perl -e 'local $/; open IN, $fn = shift @ARGV or die; $f = <IN>; map { substr $f, 1024*1024 + $_->[0], 4, pack "N", $_->[1] } ([460742, 433], [463407, 199]); open OUT, ">$fn" or die; print OUT $f' backup-2020-07-14-20-20-58.753553/kvranges/snapshot.000001480611128287/0/range,1480720638239,7f0d9e2ac30a83076f8a364e60ba2a19,1048576

satherton on Jul 31, 2020