solana: vote hash mismatch on mainter against mb

Problem

After several ~ 24 hours, master’s solana-validator forks off from mb. v1.6 branch isn’t affected:

BAD

$ grep  -A1 bank.*rozen.*7705208[10].*cap solana-validator-ALRTjMiAkxNNzTeRho1iW3hPMr9bRawicW2hdLDYe1sv.log.2021y05m06d02h30m49s419451345ns
[2021-05-06T04:30:48.681506333Z INFO  solana_runtime::bank] bank frozen: 77052080 hash: Ez7JguTesdhMeJXbCy2Pqe3MEAyTcXP9jJAhwr1dsU9c accounts_delta: Db56a62B3RTfYAPUHrYsTQhMsXYoAnUo5GxJD4QoyFbF signature_count: 1222 last_blockhash: 5LHwb6pw3N7CrTNtRoaqtJoD7jWK1EL8H1GTNBgPNtLu capitalization: 494729633919310853
[2021-05-06T04:30:48.681543930Z INFO  solana_runtime::bank] accounts hash slot: 77052080 stats: BankHashStats { num_updated_accounts: 3510, num_removed_accounts: 2, num_lamports_stored: 12250358615251674, total_data_len: 411977746, num_executable_accounts: 2 }
--
[2021-05-06T04:30:49.109197394Z INFO  solana_runtime::bank] bank frozen: 77052081 hash: 55qC5WaiTeCdz7iRep42XLWqQRUXjNh4epTuNHYtZLed accounts_delta: AoQfYPepNACVDQhrTW4FHrpKQA2zssJzFUVfVy6ZHbFW signature_count: 494 last_blockhash: C5Ff3EhB9zeUbdfiXtNpnhsoBKwhwCTwWfdn8igWXHmA capitalization: 494729633918075853
[2021-05-06T04:30:49.109220784Z INFO  solana_runtime::bank] accounts hash slot: 77052081 stats: BankHashStats { num_updated_accounts: 1759, num_removed_accounts: 5, num_lamports_stored: 7984299796894355, total_data_len: 276549138, num_executable_accounts: 3 }

GOOD

$ grep -A1 bank.*rozen.*7705208[10].*cap solana-validator-21XVxUs4s1zSgkGNWgQDvTkNwKwgAWYHwYaiPyrr3HLA.log
[2021-05-06T04:30:48.605019353Z INFO  solana_runtime::bank] bank frozen: 77052080 hash: Ez7JguTesdhMeJXbCy2Pqe3MEAyTcXP9jJAhwr1dsU9c accounts_delta: Db56a62B3RTfYAPUHrYsTQhMsXYoAnUo5GxJD4QoyFbF signature_count: 1222 last_blockhash: 5LHwb6pw3N7CrTNtRoaqtJoD7jWK1EL8H1GTNBgPNtLu capitalization: 494729633919310853
[2021-05-06T04:30:48.605051351Z INFO  solana_runtime::bank] accounts hash slot: 77052080 stats: BankHashStats { num_updated_accounts: 3510, num_removed_accounts: 2, num_lamports_stored: 12250358615251674, total_data_len: 411977746, num_executable_accounts: 2 }
--
[2021-05-06T04:30:49.115992893Z INFO  solana_runtime::bank] bank frozen: 77052081 hash: 37UhwEWabghsD14tjSjY65SMhQg4hRABu7hYCxpXfP5M accounts_delta: 4CovMT53n9x2TnfswtvGkoYFrRASnhqbnKmF9foktWUY signature_count: 494 last_blockhash: C5Ff3EhB9zeUbdfiXtNpnhsoBKwhwCTwWfdn8igWXHmA capitalization: 494729633918075853
[2021-05-06T04:30:49.116017020Z INFO  solana_runtime::bank] accounts hash slot: 77052081 stats: BankHashStats { num_updated_accounts: 1759, num_removed_accounts: 5, num_lamports_stored: 7984299796894355, total_data_len: 276549138, num_executable_accounts: 3 }

Surprisingly, logs doesn’t tell… it seems there’s very subtle difference.

Proposed Solution

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 22 (22 by maintainers)

Most upvoted comments

@ryoqun Thanks for verifying that.

We still need to figure out what situation / scenario specifically causes both of these (#17083 and #17135). Because the tests don’t cover it yet, but they need to. One thing they both seem to have in common is the UpgradeableLoader writing to an account from account_deps.

Lichtso on May 14, 2021

@ryoqun

I’m pretty sure this vote hash mismatch is exactly caused by the commit 9dfcb92… (I bisected…) Could you take a look at this as the next turn?

We have narrowed it down even further inside the PR and found the following “fix”: https://github.com/Lichtso/solana/commits/fix/invoke_context_push_account_deps

But, I am still not sure why this changes it … or if the two issues (#17083 and #17135) are actually caused by the same thing.

Lichtso on May 13, 2021

@ryoqun here and above. I’m working with @Lichtso right now on this.

jeffwashington on May 13, 2021