solana: testnet: verify_bank_hash failed: MismatchedTotalLamports

testnet.solana.com (tv) started generating invalid snapshots sometime recently. Meanwhile tw is still generating valid snapshots. Both are running

$ solana --version
solana-cli 1.5.8 (src:460c643f; feat:4077612551)

On both tw and tv, there’s an issue-15504/ directory with the a snapshot for the same slot generated on both machines. The tw version is ok, tv version is not. Logs are also in that directory. To reproduce the error, run: solana-ledger-tool -l issue-15504/ verify on both machines.

This may be related to https://github.com/solana-labs/solana/issues/15180

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (16 by maintainers)

Most upvoted comments

@carllin - you’re basically working on this so I’ll assign the issue to you

mvines on Feb 26, 2021

This was the last new root before the slot was flushed:

[2021-02-22T18:20:11.234906489Z INFO  solana_core::replay_stage] new root 64820839
[2021-02-22T18:20:11.250659016Z INFO  solana_runtime::accounts_db] flush_slot_cache slot: 64820839
[2021-02-22T18:20:11.639581110Z INFO  solana_runtime::accounts_db] flush_slot_cache slot: 64820739
[2021-02-22T18:20:11.640541096Z INFO  solana_core::replay_stage] new fork:64820880 parent:64820879 root:64820839

@carllin maybe we can replay this scenario and flush the cache in the same place and see what the live account set is for this slot?

sakridge on Feb 24, 2021

This is the difference:

key	hash	lamports	slot
TyGedS1k1ZKUrR42avUR36k9geFBzRo1QZndJ24o85K	F9Fm5eFGgy8brwqai21nSdnJpjSkbVgJeSBaBCDHfkv1	2039280	64820103

Only present in the bad snapshot.

sakridge on Feb 24, 2021