besu: Rocks DB: io.vertx.core.VertxException: Thread blocked (--Xplugin-rocksdb-high-spec-enabled)
My node failed to attest and needed a restart to recover using the new high spec enabled flag on v22.7.3 with bonsai trees.
=ETH,NET,WEB3,ADMIN --static-nodes-file=/secrets/static-nodes.json --Xplugin-rocksdb-high-spec-enabled
Frequency:
Happened once, stopped attesting completely. Had to restart service to recover.
Versions (Add all that apply)
besu/v22.7.3/linux-x86_64/openjdk-java-17
2022-09-26 16:59:13.153+00:00 | main | INFO | Besu | Using jemalloc
openjdk 17.0.4 2022-07-19
OpenJDK Runtime Environment (build 17.0.4+8-Ubuntu-120.04)
OpenJDK 64-Bit Server VM (build 17.0.4+8-Ubuntu-120.04, mixed mode, sharing)
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
Linux b1666277e36c 5.15.0-48-generic #54-Ubuntu SMP Fri Aug 26 13:26:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Additional Information (Add any of the following or anything else that may be relevant)
Server: 26 GB RAM, 18 cores, AMD
###LOGS
eth1_1 | 2022-09-26 16:52:45.661+00:00 | vert.x-eventloop-thread-3 | ERROR | RoutingContext | Unhandled exception in router
eth1_1 | 2022-09-26 16:52:45.662+00:00 | vert.x-eventloop-thread-3 | ERROR | RoutingContext | Unhandled exception in router
eth1_1 | 2022-09-26 16:52:56.409+00:00 | vertx-blocked-thread-checker | WARN | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 1640884 ms, time limit is 60000 ms
eth1_1 | io.vertx.core.VertxException: Thread blocked
eth1_1 | at app//org.rocksdb.RocksDB.get(Native Method)
eth1_1 | at app//org.rocksdb.RocksDB.get(RocksDB.java:2084)
eth1_1 | at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.get(RocksDBColumnarKeyValueStorage.java:209)
eth1_1 | at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.get(RocksDBColumnarKeyValueStorage.java:64)
eth1_1 | at app//org.hyperledger.besu.services.kvstore.SegmentedKeyValueStorageAdapter.get(SegmentedKeyValueStorageAdapter.java:50)
eth1_1 | at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage.getAccountStateTrieNode(BonsaiWorldStateKeyValueStorage.java:128)
eth1_1 | at app//org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.getAccountStateTrieNode(BonsaiPersistedWorldState.java:346)
eth1_1 | at app//org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState$$Lambda$1563/0x000000080126c548.getNode(Unknown Source)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredNodeFactory.retrieve(StoredNodeFactory.java:96)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:130)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:63)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.PutVisitor.visit(PutVisitor.java:68)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:64)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.PutVisitor.visit(PutVisitor.java:68)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:64)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.PutVisitor.visit(PutVisitor.java:68)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:64)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.PutVisitor.visit(PutVisitor.java:68)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.PutVisitor.visit(PutVisitor.java:68)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.PutVisitor.visit(PutVisitor.java:68)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.put(StoredMerklePatriciaTrie.java:142)
eth1_1 | at app//org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.calculateRootHash(BonsaiPersistedWorldState.java:224)
eth1_1 | at app//org.hyperledger.besu.ethereum.bonsai.BonsaiInMemoryWorldState.rootHash(BonsaiInMemoryWorldState.java:43)
eth1_1 | at app//org.hyperledger.besu.ethereum.bonsai.BonsaiInMemoryWorldState.persist(BonsaiInMemoryWorldState.java:53)
eth1_1 | at app//org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:220)
eth1_1 | at app//org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:43)
eth1_1 | at app//org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:80)
eth1_1 | at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:175)
eth1_1 | at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAnd
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (14 by maintainers)
Hmm interesting @ahamlat - should we limit by default? Or another behavior. Tagging @mark-terry to look at for his RPC investigation.
I just found in the heapdump fromBlock and toBlock parameters of the eth_getLog call, it is from 0x0 (0 in decimal) to 0xf180de (15827166 in decimal) which represents the whole blocks from genesis to yesterday. There is also a parameter to filter on, but I guess is less efficient with such a number of blocks to filter.