besu: Critical Exception Processing Transaction

Description

My Besu node seems to have become corrupt, possibly after a reorg?

I think this is the second time in a couple weeks this happened. I just re-sync a few weeks ago because of a similar or if not the same bug (I didn’t look into it that hard, I figured it was fluke)

2022-12-02 18:57:24.300+00:00 | vert.x-worker-thread-0 | INFO  | EngineNewPayload | Imported #16,098,404 / 156 tx / base fee 14.19 gwei / 19,826,119 (66.1%) gas / (0xabc875990fc719c8363fc652c68b84465514a4be0faf57b69e78da016bb1bc2d) in 2.467s. Peers: 17
2022-12-02 18:57:31.013+00:00 | vert.x-eventloop-thread-11 | ERROR | EthStatsService | Failed to reach the ethstats server Failed to resolve 'ethstats.net' after 2 queries 
2022-12-02 18:57:42.724+00:00 | vert.x-eventloop-thread-13 | ERROR | EthStatsService | Failed to reach the ethstats server Failed to resolve 'ethstats.net' after 2 queries 
2022-12-02 18:57:53.796+00:00 | vert.x-eventloop-thread-15 | ERROR | EthStatsService | Failed to reach the ethstats server Failed to resolve 'ethstats.net' after 2 queries 
2022-12-02 18:57:55.394+00:00 | vert.x-worker-thread-0 | WARN  | DefaultBlockchain | Chain Reorganization +9 new / -0 old
       Old - hash: 0xfd2febdf386ce960c55e6dfbbb36027da157e0b78bcbae677cd51c1b4e0cf5e6, height: 16098394
       New - hash: 0x6371917fba3322ba5341b58c20ea6b122fd6071eb9dbd57ad0d28ca5f44b626f, height: 16098403
  Ancestor - hash: 0xfd2febdf386ce960c55e6dfbbb36027da157e0b78bcbae677cd51c1b4e0cf5e6, height: 16098394
2022-12-02 18:57:55.476+00:00 | vert.x-worker-thread-0 | INFO  | EngineForkchoiceUpdated | VALID for fork-choice-update: head: 0x6371917fba3322ba5341b58c20ea6b122fd6071eb9dbd57ad0d28ca5f44b626f, finalized: 0xd585be57185dae288b2308c8360e5b2e3d2d2219c089d7e82e04c1a8f46ed6b6, safeBlockHash: 0xeab7558264de8d37727a07f432424866a7463c2699490073e25f46519f94cb84
2022-12-02 18:57:58.830+00:00 | vert.x-worker-thread-0 | ERROR | MainnetTransactionProcessor | Critical Exception Processing Transaction
org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x02e481ac0a91359eba51c88ffd254bbef048e6a71c810c38cba58a68146befbb location 0x
	at org.hyperledger.besu.ethereum.trie.StoredNode.lambda$load$0(StoredNode.java:133)
	at java.base/java.util.Optional.orElseThrow(Optional.java:408)
	at org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:131)
	at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:63)
	at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:119)
	at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage.getStorageValueBySlotHash(BonsaiWorldStateKeyValueStorage.java:199)
	at org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.getStorageValueBySlotHash(BonsaiPersistedWorldState.java:421)
	at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getStorageValueBySlotHash(BonsaiWorldStateUpdater.java:353)
	at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getStorageValue(BonsaiWorldStateUpdater.java:340)
	at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.getStorageValue(BonsaiAccount.java:208)
	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
	at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:213)
	at org.hyperledger.besu.evm.operation.SLoadOperation.execute(SLoadOperation.java:58)
	at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:282)
	at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:161)
	at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:173)
	at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:492)
	at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:404)
	at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:104)
	at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:76)
	at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:205)
	at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
	at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:440)
	at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:454)
	at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:141)
	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.EngineNewPayload.syncResponse(EngineNewPayload.java:217)
	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022-12-02 18:57:58.831+00:00 | vert.x-worker-thread-0 | INFO  | AbstractBlockProcessor | Block processing error: transaction invalid Internal Error in Besu - org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x02e481ac0a91359eba51c88ffd254bbef048e6a71c810c38cba58a68146befbb location 0x. Block 0x47d61f6dca45a190d79b2f5fda11c0b8eca83ce1ba2a5c8675e3d37eda836ac1 Transaction 0x7a74c82d2174681c6cd1cc529d685cd76f4fc470967cf107b503a151eecea617
2022-12-02 18:57:58.832+00:00 | vert.x-worker-thread-0 | INFO  | MainnetBlockValidator | Optional[Block processing error: transaction invalid Internal Error in Besu - org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x02e481ac0a91359eba51c88ffd254bbef048e6a71c810c38cba58a68146befbb location 0x. Block 0x47d61f6dca45a190d79b2f5fda11c0b8eca83ce1ba2a5c8675e3d37eda836ac1 Transaction 0x7a74c82d2174681c6cd1cc529d685cd76f4fc470967cf107b503a151eecea617]. Block 16098405 (0x47d61f6dca45a190d79b2f5fda11c0b8eca83ce1ba2a5c8675e3d37eda836ac1)
2022-12-02 18:57:58.836+00:00 | vert.x-worker-thread-0 | WARN  | EngineNewPayload | Invalid new payload: number: 16098405, hash: 0x47d61f6dca45a190d79b2f5fda11c0b8eca83ce1ba2a5c8675e3d37eda836ac1, parentHash: 0xabc875990fc719c8363fc652c68b84465514a4be0faf57b69e78da016bb1bc2d, latestValidHash: 0xabc875990fc719c8363fc652c68b84465514a4be0faf57b69e78da016bb1bc2d, status: INVALID, validationError: Block processing error: transaction invalid Internal Error in Besu - org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x02e481ac0a91359eba51c88ffd254bbef048e6a71c810c38cba58a68146befbb location 0x. Block 0x47d61f6dca45a190d79b2f5fda11c0b8eca83ce1ba2a5c8675e3d37eda836ac1 Transaction 0x7a74c82d2174681c6cd1cc529d685cd76f4fc470967cf107b503a151eecea617

Here is the important part of my Docker-Compose:

    image: hyperledger/besu:latest
    command:
      - --data-path=/data
      - --network=MAINNET
      - --data-storage-format=BONSAI
      - --Xbonsai-use-snapshots=true
      - --sync-mode=X_SNAP
      - --p2p-port=30303
      - --pruning-enabled
      - --nat-method=DOCKER
      - --rpc-http-enabled=true
      - --rpc-http-apis=ADMIN,DEBUG,ETH,NET,TRACE,TXPOOL,WEB3
      - --rpc-http-host=0.0.0.0
      - --rpc-http-port=8545
      - --rpc-http-cors-origins=*
      - --host-allowlist=*
      - --rpc-ws-enabled=true
      - --rpc-ws-apis=ADMIN,DEBUG,ETH,NET,TRACE,TXPOOL,WEB3
      - --rpc-ws-host=0.0.0.0
      - --rpc-ws-port=8546
      - --engine-rpc-enabled
      - --engine-rpc-port=8551
      - --engine-jwt-secret=/data/jwtsecret
      - --engine-host-allowlist=*
      - --Xplugin-rocksdb-high-spec-enabled

I also have some of these in my logs from earlier:

2022-12-02 16:49:17.231+00:00 | vert.x-worker-thread-0 | WARN  | EngineNewPayload | Invalid new payload: number: 16098395, hash: 0x3d23c3ff2ab9cdb7dec49c0314660c93c6f65b5607f716f6f15d00798c1a281e, parentHash: 0xfd2febdf386ce960c55e6dfbbb36027da157e0b78bcbae677cd51c1b4e0cf5e6, latestValidHash: 0x0000000000000000000000000000000000000000000000000000000000000000, status: INVALID, validationError: Block already present in bad block manager.