bitcoin: salvagewallet fails verification
This is somewhat rare but still worth to take a look sometime. I can reproduce locally and travis complains in wallet.py:
check -salvagewallet
start_node: bitcoind started, calling bitcoin-cli -rpcwait getblockcount
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
Edit:
This will now show up as
check -salvagewallet
Unexpected exception caught during testing: Exception('bitcoind exited with status 1 during initialization',)
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to laanwj/bitcoin by laanwj 8 years ago
- test_framework: detect failure of bitcoind startup Replace the `bitcoin-cli -rpcwait` after spawning bitcoind with our own loop that detects when bitcoind exits prematurely. And if one node fails to... — committed to laanwj/bitcoin by laanwj 8 years ago
- cherry-pick Core PR#7744 to abort test instead of freezing up This is a merge of bitcoin/bitcoin#7744 / MVF-Core PR#16. Instead of freezing up qa tests, wallet.py will fail on some 32-bit platforms ... — committed to ftrader-bitcoinunlimited/hardfork_prototype_1_mvf-bu by ftrader 8 years ago
- Disable salvagewallet in GUI (#1390) * Disable salvagewallet in GUI salvagewallet is known to cause problems and it's way to powerful to let it be accessible in GUI https://github.com/bitcoin/bi... — committed to dashpay/dash by UdjinM6 7 years ago
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to UdjinM6/dash by laanwj 8 years ago
- Disable salvagewallet in GUI (#1390) * Disable salvagewallet in GUI salvagewallet is known to cause problems and it's way to powerful to let it be accessible in GUI https://github.com/bitcoin/bitcoi... — committed to AmirolAhmad/gobytes-test by UdjinM6 7 years ago
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to lateminer/bitcoin by laanwj 8 years ago
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to furszy/bitcoin-core by laanwj 8 years ago
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to furszy/bitcoin-core by laanwj 8 years ago
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to furszy/bitcoin-core by laanwj 8 years ago
- wallet: Warn on unexpected EOF while salvaging wallet Check for EOF before every getline, and warn when reading gets to EOF before the end of the data. Stricter error checking could shed more light ... — committed to furszy/bitcoin-core by laanwj 8 years ago
I’ve been digging into this failure over the last couple of days. This is what I’ve found so far:
This is fairly easy to reproduce. Just uncomment the
-salvagewallettest and set BASE_SCRIPTS in test_runner.py to be a list[["wallet.py"] * 50]. In a run of 50, I usually get one failure.the problem is caused by running Berkeley DB’s
Db::verify()with theDB_AGGRESSIVEflag. The documentation for that is (emphasis mine):(from the BDB 4.8.30 docs)
Running
DB::verify()with agressive set causes the output to be corrupt (more details below). I’ve tried this with the most recent version of Berkeley DB from http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html and an aggressive verify still corrupts the database.you can repro the corruption manually by running the
db_dumputility function, which is bundled with BDB.-rruns salvage and-Rruns salvage with aggressive.This is the wallet.dat file that I was using in my tests (renamed wallet.dat.txt as github has rules about file types): wallet.dat.txt. If I run db_dump on this .dat file with -R, the output ends:
if I run
db_dumpwith-r, then I don’t see theDB_VERIFY_BADmessage.piping
db_dumpthrough todb_loadrebuilds a database. If I use-Ron db_dump, then this fails, because the database dump is corrupt. If I use-r, on db_dump, then it succeeds, and most (all?) of the key-value pairs appear to be intact.Digging a bit more into the corruption, if I look at the output of
db_dump -R, it’s a superset of the output ofdb_dump -r. There are some extra rows that look like normal keys/values, but there are also a bunch of these:554e4b4e4f574e5f4b4559is ascii forUNKNOWN_KEY. You can see where that string is output in btree/bt_verify.c in the BDB source.the output format for BDB should be key in one row followed by value in the next row. You can see here that the final
554e4b4e4f574e5f4b4559key is followed by036b65792103c58cdbfc2b4cedd96e56d2714aa980a1478feb0bc5f9f7b6036e075eb974faef, which is actually another key row (for a bitcoin public key). That means that the output of db_dump is out of alignment - there’s actually an odd number of rows, so after this line, parsing keys/values gets mixed up.ReadKeyValue()in bitcoind and see exactly the same thing when running with-salvagewallet. The key/values get misaligned when re-reading after callingDb::verify():(you can see at the end that it’s reading keys as values and vice versa).
After running with
-salvagewallet, thewallet.datfile is empty. All key-value pairs have gone. The wallet backup filewallet.<timestamp>.bakcontains the db from before the salvage operation so still has all the key-value pairs.Next steps:
-salvagewalletshould be changed to not set the aggressive flag. Note the documentation for that flag: the output will almost certainly require editing before being loaded into a database. Currently the salvagewallet operation just directly reads that output, so it’s not entirely surprising that it occasionally fails.Db::verify()with theDB_AGGRESSIVEflag fails. Additional logging will help there. I can also add an option to the integration tests which verifies the wallet.dat files of every node at the end of every test.You can just run
and come back 20 minutes later to dig through the dumps.