solana: Consensus (bank hash) mismatch between v1.8.10 and tip of master/v1.9.1 (mainnet)

Problem

A node running master (commit 94aa9e568a04cf66fadd05275c5b9728ea6b55d1) against mainnet was found to have a consensus failure; this occurred in slot 111883235:

[2021-12-17T19:36:41.237049780Z INFO  solana_ledger_tool] solana-ledger-tool 1.8.10 (src:f97d28ca; feat:52865992)
...
[2021-12-17T19:38:18.800288320Z INFO  solana_runtime::bank]
bank frozen: 111883235
hash: 4JTffyL2mXd17GXRDCBbUSg5FD2h6BUsCBobf7ZUer9g 
accounts_delta: 396ZSKNKJhth9CmSte7Lyf8HS5vsd3TJLNcEost6VHZX
signature_count: 753
last_blockhash: H9uGK6rZVsXG8J6akmJ5oyXczUR22rRnLu4UGBw1wfJ9
capitalization: 511214148638373382
[2021-12-17T19:38:18.800317947Z INFO  solana_runtime::bank]
accounts hash slot: 111883235
stats: BankHashStats {
    num_updated_accounts: 3466,
    num_removed_accounts: 8,
    num_lamports_stored: 11203702455212782,
    total_data_len: 115749862,
    num_executable_accounts: 1 }

[2021-12-17T19:47:53.521219209Z INFO  solana_ledger_tool] solana-ledger-tool 1.10.0 (src:3dab1e71; feat:3834082194)
[2021-12-17T19:49:11.369273370Z INFO  solana_runtime::bank]
bank frozen: 111883235
hash: CYvAB1HULgZQHmKbYAnRrCEMJ2jRju3UsSq7U6GQZ3FU
accounts_delta: 7mrWSpzRjRuDWXp84iu37CptBAZGVxJbzznJg64dJEM3
signature_count: 753
last_blockhash: H9uGK6rZVsXG8J6akmJ5oyXczUR22rRnLu4UGBw1wfJ9
capitalization: 511214148638373382
[2021-12-17T19:49:11.369305502Z INFO  solana_runtime::bank]
accounts hash slot: 111883235
stats: BankHashStats {
    num_updated_accounts: 3463,
    num_removed_accounts: 7,
    num_lamports_stored: 11203689947327182,
    total_data_len: 113953143,
    num_executable_accounts: 0

Here is the offending transaction:

  Entry 629 - num_hashes: 4225, hash: 4ypLNZyp1k3DLtfGCwb616RHiUSdLH2mvBcpgixiM1nd, transactions: 1
    Transaction 0
      Recent Blockhash: 6MVtZba5uYYcsmmnYv3Yht4XVQy43H7X285wbLXe66ge
      Signature 0: t4nmKNdkLSQTMvxGj9JRV29Ff7stD1okMGiens9gH3wEkNVzRB3NmwSFs6FCbX9dWuxUZU6wrKfV3UKptiZbFNU
      Account 0: srw- 3rWmJZgB2wtSsYCBYouL4mYRBZhH9V363BVs7HBaJUy9 (fee payer)
      Account 1: -rw- 65dX1iH5k59ganpRL1sdgVdGQSyVVXwPES1voMuoRG3x
      Account 2: -rw- 4eaHV6yMLTv2oj1zq64H9JcVaZyUu8xDiSpRuF5p2BAa
      Account 3: -rw- 7ek4EpFaoForoFzzx8wBVQzN9oYP5reNK9JkYHMnTxo9
      Account 4: -rwx wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb
      Account 5: -r-- yKkPofCpS7TapAnJxGYMan8QtNAurzXeisYKg2rhJWp
      Account 6: -r-- DHyAcRbFpRWTkcsAsfwQpbABXvtjs6bQ1dq5ScNhRDoQ
      Account 7: -r-- SysvarRent111111111111111111111111111111111
      Account 8: -r-- SysvarC1ock11111111111111111111111111111111
      Account 9: -r-- BPFLoaderUpgradeab1e11111111111111111111111
      Account 10: -r-- 11111111111111111111111111111111
      Instruction 0
        Program:   wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb (4)
        Account 0: 3rWmJZgB2wtSsYCBYouL4mYRBZhH9V363BVs7HBaJUy9 (0)
        Account 1: yKkPofCpS7TapAnJxGYMan8QtNAurzXeisYKg2rhJWp (5)
        Account 2: 65dX1iH5k59ganpRL1sdgVdGQSyVVXwPES1voMuoRG3x (1)
        Account 3: DHyAcRbFpRWTkcsAsfwQpbABXvtjs6bQ1dq5ScNhRDoQ (6)
        Account 4: 3rWmJZgB2wtSsYCBYouL4mYRBZhH9V363BVs7HBaJUy9 (0)
        Account 5: 4eaHV6yMLTv2oj1zq64H9JcVaZyUu8xDiSpRuF5p2BAa (2)
        Account 6: 7ek4EpFaoForoFzzx8wBVQzN9oYP5reNK9JkYHMnTxo9 (3)
        Account 7: wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb (4)
        Account 8: SysvarRent111111111111111111111111111111111 (7)
        Account 9: SysvarC1ock11111111111111111111111111111111 (8)
        Account 10: BPFLoaderUpgradeab1e11111111111111111111111 (9)
        Account 11: 11111111111111111111111111111111 (10)
        Data: [8]

And some logging output about this transaction:

[2021-12-17T20:19:42.381621999Z DEBUG solana_program_runtime::stable_log] Program wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb invoke [1]
[2021-12-17T20:19:42.381782384Z DEBUG solana_program_runtime::stable_log] Program 11111111111111111111111111111111 invoke [2]
[2021-12-17T20:19:42.381795339Z DEBUG solana_program_runtime::stable_log] Program 11111111111111111111111111111111 success
[2021-12-17T20:19:42.381840094Z DEBUG solana_program_runtime::invoke_context] wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb's writable privilege escalated
[2021-12-17T20:19:42.381844743Z DEBUG solana_bpf_loader_program] Program wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb consumed 48494 of 200000 compute units
[2021-12-17T20:19:42.381850574Z DEBUG solana_program_runtime::stable_log] Program wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb failed: Cross-program invocation with unauthorized signer or writable account
[2021-12-17T20:19:42.382633506Z DEBUG solana_runtime::bank] check: 1us load: 578us execute: 20111us txs_len=1
[2021-12-17T20:19:42.382638044Z DEBUG solana_runtime::bank] tx error: Err(InstructionError(0, PrivilegeEscalation))
SanitizedTransaction {
    message: Legacy(Message {
        header: MessageHeader {
            num_required_signatures: 1,
            num_readonly_signed_accounts: 0,
            num_readonly_unsigned_accounts: 6 },
        account_keys: [3rWmJZgB2wtSsYCBYouL4mYRBZhH9V363BVs7HBaJUy9,
                       65dX1iH5k59ganpRL1sdgVdGQSyVVXwPES1voMuoRG3x,
                       4eaHV6yMLTv2oj1zq64H9JcVaZyUu8xDiSpRuF5p2BAa,
                       7ek4EpFaoForoFzzx8wBVQzN9oYP5reNK9JkYHMnTxo9,
                       wormDTUJ6AWPNvk59vGQbDvGJmqbDTdgWgAqcLBCgUb,
                       yKkPofCpS7TapAnJxGYMan8QtNAurzXeisYKg2rhJWp,
                       DHyAcRbFpRWTkcsAsfwQpbABXvtjs6bQ1dq5ScNhRDoQ,
                       SysvarRent111111111111111111111111111111111,
                       SysvarC1ock11111111111111111111111111111111,
                       BPFLoaderUpgradeab1e11111111111111111111111,
                       11111111111111111111111111111111],
        recent_blockhash: 6MVtZba5uYYcsmmnYv3Yht4XVQy43H7X285wbLXe66ge,
        instructions: [CompiledInstruction {
            program_id_index: 4,
            accounts: [0, 5, 1, 6, 0, 2, 3, 4, 7, 8, 9, 10], data: [8] }] }),
            message_hash: ArogkuF3r2wci1GHDLfHVihJAj9tAxUAExznE7UfcFmn,
            is_simple_vote_tx: false,
            signatures: [t4nmKNdkLSQTMvxGj9JRV29Ff7stD1okMGiens9gH3wEkNVzRB3NmwSFs6FCbX9dWuxUZU6wrKfV3UKptiZbFNU] }
[2021-12-17T20:19:42.382679443Z DEBUG solana_runtime::bank] 1 errors of 1 txs

Steps to Reproduce

1. Pull down and expand this ledger
2. Run solana-ledger-tool verify --ledger ~/21986_ledger and note the bank frozen: 111883235 ... line of output