besu: Mainnet node desyncs: World State Root does not match expected value

Description

Mainnet besu node that was in sync has stopped adding blocks. It is still active and receiving blocks, but throwing them out with a “World State Root does not match expected value” error.

As a side result of this, besu is kicking out all of the peers that disagree with it.

Confirmed that head block is stuck at the point the errors began (head at time of writing is 14833965):

$ # curl -s http://localhost:8545 -X POST -H "Content-Type: application/json" --data '{"method":"eth_getBlockByNumber","params":["latest", true],"id":1,"jsonrpc":"2.0"}'| jq -r '.result.number' | xargs printf '%d\n'
14833808

Acceptance Criteria

Besu should continue to process blocks and build the chain.

Versions (Add all that apply)

$ besu --version
besu/v22.4.0/linux-x86_64/openjdk-java-17

$ java --version
openjdk 17.0.3 2022-04-19
OpenJDK Runtime Environment (build 17.0.3+7-Ubuntu-0ubuntu0.20.04.1)
OpenJDK 64-Bit Server VM (build 17.0.3+7-Ubuntu-0ubuntu0.20.04.1, mixed mode, sharing)

This is using data-storage-format="bonsai"

Logs

The initial error was:

2022-05-24 04:57:07.909+00:00 | EthScheduler-Workers-2 | INFO  | PersistBlockTask | Imported #14,833,806 / 125 tx / 0 om / 5,371,865 (17.9%) gas / (0x582cfeb634fc709df209a666ed37d882ad06da4595f091391a2a53c2e4c3be62) in 0.177s. Peers: 30
2022-05-24 04:57:11.122+00:00 | EthScheduler-Workers-1 | INFO  | PersistBlockTask | Imported #14,833,807 / 274 tx / 0 om / 29,996,473 (100.0%) gas / (0x4526e1abe05e04857f13296cbfb7b3143fd182ec85141b4b342d44de5d8baf43) in 0.813s. Peers: 30
2022-05-24 04:57:14.045+00:00 | EthScheduler-Workers-2 | INFO  | PersistBlockTask | Imported #14,833,808 / 154 tx / 0 om / 11,912,845 (39.7%) gas / (0xb0fb5182a37031d3d747a70ee63c71bee73e613ce07e4bb215cba9df974e4ed1) in 0.300s. Peers: 30
2022-05-24 04:57:36.166+00:00 | EthScheduler-Workers-0 | ERROR | AbstractBlockProcessor | failed persisting block
java.lang.RuntimeException: World State Root does not match expected value, header 0xa6501affc39a24c0c46e70973f9e34b16c6c090d1d7851dba98de0cb51a506c4 calculated 0x6d8c8d327044fd196b3beaa453a403958c717693300b660072a40157512603ab
        at org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.persist(BonsaiPersistedWorldState.java:264)
        at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:203)
        at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:42)
        at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:75)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:146)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:100)
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:44)
        at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:44)
        at org.hyperledger.besu.ethereum.eth.sync.tasks.PersistBlockTask.executeTask(PersistBlockTask.java:201)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractEthTask.executeTaskTimed(AbstractEthTask.java:150)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractEthTask.run(AbstractEthTask.java:75)
        at org.hyperledger.besu.ethereum.eth.sync.BlockPropagationManager.runImportTask(BlockPropagationManager.java:482)
        at org.hyperledger.besu.ethereum.eth.sync.BlockPropagationManager.validateAndProcessPendingBlock(BlockPropagationManager.java:463)
        at org.hyperledger.besu.ethereum.eth.sync.BlockPropagationManager.lambda$importOrSavePendingBlock$17(BlockPropagationManager.java:451)
        at org.hyperledger.besu.util.FutureUtils.propagateResult(FutureUtils.java:97)
        at org.hyperledger.besu.ethereum.eth.manager.EthScheduler.lambda$scheduleSyncWorkerTask$0(EthScheduler.java:108)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
2022-05-24 04:57:36.166+00:00 | EthScheduler-Workers-0 | ERROR | MainnetBlockValidator | Error processing block. Block 14833809 (0x15ab57cc5bd28eb54a1bfc45d49e182420b4a8d0222c375b4bc1425072823d7e)
2022-05-24 04:57:36.166+00:00 | EthScheduler-Workers-0 | WARN  | BlockPropagationManager | Failed to import announced block 14833809 (0x15ab57cc5bd28eb54a1bfc45d49e182420b4a8d0222c375b4bc1425072823d7e)
2022-05-24 04:57:40.498+00:00 | Timer-0 | INFO  | DNSTimerTask | Refreshing DNS records with enrtree://AKA3AM6LPBYEUDMVNU3BSVQJ5AD45Y7YPOHJLEF6W26QOE4VTUDPE@all.mainnet.ethdisco.net
2022-05-24 04:57:40.985+00:00 | nioEventLoopGroup-3-1 | INFO  | BlockPropagationManager | Saving announced block 14833810 (0x3fda3e8a703029f6224c588a8dbc672bb2a084d9ecc1bd4e3e5f2837543d7ec8) for future import
2022-05-24 04:57:45.111+00:00 | nioEventLoopGroup-3-1 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x170d3457208e4e1842... PeerReputation 100, validated? true, disconnected? false at block 14833808
2022-05-24 04:57:45.431+00:00 | nioEventLoopGroup-3-3 | INFO  | BlockPropagationManager | Saving announced block 14833811 (0x3fc66bace399c0f24dad7bc7fba9badf382a455cd75caf5a2857406321d594e4) for future import
2022-05-24 04:57:46.178+00:00 | EthScheduler-Services-318 (importBlock) | ERROR | AbstractBlockProcessor | failed persisting block
java.lang.RuntimeException: World State Root does not match expected value, header 0xa6501affc39a24c0c46e70973f9e34b16c6c090d1d7851dba98de0cb51a506c4 calculated 0x6d8c8d327044fd196b3beaa453a403958c717693300b660072a40157512603ab
        at org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.persist(BonsaiPersistedWorldState.java:264)
        at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:203)
        at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:42)
        at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:75)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:146)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:100)
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:44)
        at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:44)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:61)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:31)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:152)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
2022-05-24 04:57:46.178+00:00 | EthScheduler-Services-318 (importBlock) | ERROR | MainnetBlockValidator | Error processing block. Block 14833809 (0x15ab57cc5bd28eb54a1bfc45d49e182420b4a8d0222c375b4bc1425072823d7e)
2022-05-24 04:57:46.178+00:00 | EthScheduler-Services-318 (importBlock) | WARN  | PipelineChainDownloader | Invalid block detected. Disconnecting from sync target. Failed to import block: Invalid block at #14833809 (0x15ab57cc5bd28eb54a1bfc45d49e182420b4a8d0222c375b4bc1425072823d7e)
2022-05-24 04:57:46.178+00:00 | EthScheduler-Services-318 (importBlock) | WARN  | PipelineChainDownloader | Chain download failed. Restarting after short delay.
java.util.concurrent.CompletionException: org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block at #14833809 (0x15ab57cc5bd28eb54a1bfc45d49e182420b4a8d0222c375b4bc1425072823d7e)
        at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
...

The error repeats for each block received.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (16 by maintainers)

Most upvoted comments

@matkt yes (see log below; let me know if you just want the entire log somewhere).

@jflo plain java process.

$ egrep '(FastSyncDownloader|Pivot)' besu.log
2022-05-21 09:55:47.662+00:00 | main | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 09:55:47.662+00:00 | main | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 09:58:32.295+00:00 | EthScheduler-Timer-0 | INFO  | PivotSelectorFromPeers | Selecting block number 14816537 as fast sync pivot block.
2022-05-21 09:58:32.297+00:00 | EthScheduler-Timer-0 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 09:58:32.298+00:00 | EthScheduler-Timer-0 | INFO  | PivotBlockConfirmer | Confirm pivot block 14816537 with at least 5 peers.
2022-05-21 09:58:32.456+00:00 | nioEventLoopGroup-3-3 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14816537: 0xfda5ac00fd759ed7703d54600b11e435f0d53f36663cacfb64e1f22c4c6c3c64
2022-05-21 09:58:32.458+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14816537: 0xfda5ac00fd759ed7703d54600b11e435f0d53f36663cacfb64e1f22c4c6c3c64
2022-05-21 09:58:32.495+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14816537: 0xfda5ac00fd759ed7703d54600b11e435f0d53f36663cacfb64e1f22c4c6c3c64
2022-05-21 10:00:55.584+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14816537: 0xfda5ac00fd759ed7703d54600b11e435f0d53f36663cacfb64e1f22c4c6c3c64
2022-05-21 10:01:02.048+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14816537: 0xfda5ac00fd759ed7703d54600b11e435f0d53f36663cacfb64e1f22c4c6c3c64
2022-05-21 13:18:16.275+00:00 | nioEventLoopGroup-3-9 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-21 13:18:16.276+00:00 | nioEventLoopGroup-3-9 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 13:18:16.276+00:00 | nioEventLoopGroup-3-9 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 13:18:16.518+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotSelectorFromPeers | Selecting block number 14817417 as fast sync pivot block.
2022-05-21 13:18:16.518+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 13:18:16.519+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Confirm pivot block 14817417 with at least 5 peers.
2022-05-21 13:18:16.540+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14817417: 0x0f60d215db45036d4d60bed05322b781e47ed869cd81f0a8521b1c111ba09ec5
2022-05-21 13:18:16.545+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14817417: 0x0f60d215db45036d4d60bed05322b781e47ed869cd81f0a8521b1c111ba09ec5
2022-05-21 13:18:16.548+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14817417: 0x0f60d215db45036d4d60bed05322b781e47ed869cd81f0a8521b1c111ba09ec5
2022-05-21 13:18:16.616+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14817417: 0x0f60d215db45036d4d60bed05322b781e47ed869cd81f0a8521b1c111ba09ec5
2022-05-21 13:18:16.636+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14817417: 0x0f60d215db45036d4d60bed05322b781e47ed869cd81f0a8521b1c111ba09ec5
2022-05-21 14:57:47.203+00:00 | nioEventLoopGroup-3-7 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-21 14:57:47.211+00:00 | nioEventLoopGroup-3-7 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 14:57:47.211+00:00 | nioEventLoopGroup-3-7 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 14:57:47.475+00:00 | nioEventLoopGroup-3-7 | INFO  | PivotSelectorFromPeers | Selecting block number 14817849 as fast sync pivot block.
2022-05-21 14:57:47.476+00:00 | nioEventLoopGroup-3-7 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 14:57:47.476+00:00 | nioEventLoopGroup-3-7 | INFO  | PivotBlockConfirmer | Confirm pivot block 14817849 with at least 5 peers.
2022-05-21 14:57:47.505+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14817849: 0xd06ab12022afd3f0894a39b06b67f28b5767a10adc2798c67d8ad17065d87cbd
2022-05-21 14:57:47.561+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14817849: 0xd06ab12022afd3f0894a39b06b67f28b5767a10adc2798c67d8ad17065d87cbd
2022-05-21 14:57:47.574+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14817849: 0xd06ab12022afd3f0894a39b06b67f28b5767a10adc2798c67d8ad17065d87cbd
2022-05-21 14:57:47.588+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14817849: 0xd06ab12022afd3f0894a39b06b67f28b5767a10adc2798c67d8ad17065d87cbd
2022-05-21 14:57:47.608+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14817849: 0xd06ab12022afd3f0894a39b06b67f28b5767a10adc2798c67d8ad17065d87cbd
2022-05-21 16:06:55.933+00:00 | nioEventLoopGroup-3-1 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-21 16:06:55.935+00:00 | nioEventLoopGroup-3-1 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 16:06:55.935+00:00 | nioEventLoopGroup-3-1 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 16:06:56.031+00:00 | nioEventLoopGroup-3-1 | INFO  | PivotSelectorFromPeers | Selecting block number 14818143 as fast sync pivot block.
2022-05-21 16:06:56.032+00:00 | nioEventLoopGroup-3-1 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 16:06:56.032+00:00 | nioEventLoopGroup-3-1 | INFO  | PivotBlockConfirmer | Confirm pivot block 14818143 with at least 5 peers.
2022-05-21 16:06:56.040+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14818143: 0x8d644896e6c1a05f514396eb0259ca880f1eb6152fbe81d08a860a3f59309664
2022-05-21 16:06:56.042+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14818143: 0x8d644896e6c1a05f514396eb0259ca880f1eb6152fbe81d08a860a3f59309664
2022-05-21 16:06:56.059+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14818143: 0x8d644896e6c1a05f514396eb0259ca880f1eb6152fbe81d08a860a3f59309664
2022-05-21 16:06:56.136+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14818143: 0x8d644896e6c1a05f514396eb0259ca880f1eb6152fbe81d08a860a3f59309664
2022-05-21 16:06:56.144+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14818143: 0x8d644896e6c1a05f514396eb0259ca880f1eb6152fbe81d08a860a3f59309664
2022-05-21 19:38:13.132+00:00 | nioEventLoopGroup-3-8 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-21 19:38:13.136+00:00 | nioEventLoopGroup-3-8 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 19:38:13.136+00:00 | nioEventLoopGroup-3-8 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 19:38:13.353+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotSelectorFromPeers | Selecting block number 14819071 as fast sync pivot block.
2022-05-21 19:38:13.353+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 19:38:13.354+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockConfirmer | Confirm pivot block 14819071 with at least 5 peers.
2022-05-21 19:38:13.365+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14819071: 0xfc3098ec7bcea9946753b8c3e4eb55782c9fd63404974f152faa268e8a751980
2022-05-21 19:38:13.373+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14819071: 0xfc3098ec7bcea9946753b8c3e4eb55782c9fd63404974f152faa268e8a751980
2022-05-21 19:38:13.382+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14819071: 0xfc3098ec7bcea9946753b8c3e4eb55782c9fd63404974f152faa268e8a751980
2022-05-21 19:38:13.439+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14819071: 0xfc3098ec7bcea9946753b8c3e4eb55782c9fd63404974f152faa268e8a751980
2022-05-21 19:38:13.514+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14819071: 0xfc3098ec7bcea9946753b8c3e4eb55782c9fd63404974f152faa268e8a751980
2022-05-21 20:30:14.301+00:00 | nioEventLoopGroup-3-2 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-21 20:30:14.301+00:00 | nioEventLoopGroup-3-2 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 20:30:14.301+00:00 | nioEventLoopGroup-3-2 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 20:30:14.523+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotSelectorFromPeers | Selecting block number 14819286 as fast sync pivot block.
2022-05-21 20:30:14.557+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 20:30:14.559+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockConfirmer | Confirm pivot block 14819286 with at least 5 peers.
2022-05-21 20:30:14.564+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14819286: 0x8148fad4a8e60a6aebdd76bbc735670cc170ef731d88d502c016fe5d2c161b6f
2022-05-21 20:30:14.580+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14819286: 0x8148fad4a8e60a6aebdd76bbc735670cc170ef731d88d502c016fe5d2c161b6f
2022-05-21 20:30:14.587+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14819286: 0x8148fad4a8e60a6aebdd76bbc735670cc170ef731d88d502c016fe5d2c161b6f
2022-05-21 20:30:14.588+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14819286: 0x8148fad4a8e60a6aebdd76bbc735670cc170ef731d88d502c016fe5d2c161b6f
2022-05-21 20:30:14.645+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14819286: 0x8148fad4a8e60a6aebdd76bbc735670cc170ef731d88d502c016fe5d2c161b6f
2022-05-21 22:40:34.741+00:00 | nioEventLoopGroup-3-6 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-21 22:40:34.741+00:00 | nioEventLoopGroup-3-6 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-21 22:40:34.741+00:00 | nioEventLoopGroup-3-6 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-21 22:40:35.143+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotSelectorFromPeers | Selecting block number 14819829 as fast sync pivot block.
2022-05-21 22:40:35.155+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-21 22:40:35.175+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Confirm pivot block 14819829 with at least 5 peers.
2022-05-21 22:40:35.212+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14819829: 0x1a786148a73ce2b98e7b238acf55f94809659f02673008c6b15070cb42dc434b
2022-05-21 22:40:35.212+00:00 | nioEventLoopGroup-3-3 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14819829: 0x1a786148a73ce2b98e7b238acf55f94809659f02673008c6b15070cb42dc434b
2022-05-21 22:40:35.231+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14819829: 0x1a786148a73ce2b98e7b238acf55f94809659f02673008c6b15070cb42dc434b
2022-05-21 22:40:35.286+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14819829: 0x1a786148a73ce2b98e7b238acf55f94809659f02673008c6b15070cb42dc434b
2022-05-21 22:40:35.297+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14819829: 0x1a786148a73ce2b98e7b238acf55f94809659f02673008c6b15070cb42dc434b
2022-05-22 03:04:21.098+00:00 | nioEventLoopGroup-3-2 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-22 03:04:21.098+00:00 | nioEventLoopGroup-3-2 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-22 03:04:21.098+00:00 | nioEventLoopGroup-3-2 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-22 03:04:21.298+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotSelectorFromPeers | Selecting block number 14820973 as fast sync pivot block.
2022-05-22 03:04:21.298+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-22 03:04:21.298+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockConfirmer | Confirm pivot block 14820973 with at least 5 peers.
2022-05-22 03:04:21.300+00:00 | nioEventLoopGroup-3-3 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14820973: 0x158d3092f86bcecb5f250c7a30bda654019240e326f3331e5693f0133a7c5a65
2022-05-22 03:04:21.302+00:00 | nioEventLoopGroup-3-1 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14820973: 0x158d3092f86bcecb5f250c7a30bda654019240e326f3331e5693f0133a7c5a65
2022-05-22 03:04:21.307+00:00 | nioEventLoopGroup-3-6 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14820973: 0x158d3092f86bcecb5f250c7a30bda654019240e326f3331e5693f0133a7c5a65
2022-05-22 03:04:21.316+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14820973: 0x158d3092f86bcecb5f250c7a30bda654019240e326f3331e5693f0133a7c5a65
2022-05-22 03:04:22.036+00:00 | nioEventLoopGroup-3-10 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14820973: 0x158d3092f86bcecb5f250c7a30bda654019240e326f3331e5693f0133a7c5a65
2022-05-22 04:19:41.896+00:00 | nioEventLoopGroup-3-8 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-22 04:19:41.896+00:00 | nioEventLoopGroup-3-8 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-22 04:19:41.896+00:00 | nioEventLoopGroup-3-8 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-22 04:19:42.013+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotSelectorFromPeers | Selecting block number 14821289 as fast sync pivot block.
2022-05-22 04:19:42.014+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-22 04:19:42.014+00:00 | nioEventLoopGroup-3-8 | INFO  | PivotBlockConfirmer | Confirm pivot block 14821289 with at least 5 peers.
2022-05-22 04:19:42.017+00:00 | nioEventLoopGroup-3-1 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14821289: 0x19331fb413c12c9bc92f7916431f28ce037b98c91b93283ff8318ac255c3b57d
2022-05-22 04:19:42.020+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14821289: 0x19331fb413c12c9bc92f7916431f28ce037b98c91b93283ff8318ac255c3b57d
2022-05-22 04:19:42.041+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14821289: 0x19331fb413c12c9bc92f7916431f28ce037b98c91b93283ff8318ac255c3b57d
2022-05-22 04:19:42.113+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14821289: 0x19331fb413c12c9bc92f7916431f28ce037b98c91b93283ff8318ac255c3b57d
2022-05-22 04:19:42.191+00:00 | nioEventLoopGroup-3-3 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14821289: 0x19331fb413c12c9bc92f7916431f28ce037b98c91b93283ff8318ac255c3b57d
2022-05-22 05:24:22.030+00:00 | nioEventLoopGroup-3-10 | INFO  | FastSyncDownloader | Re-pivoting to newer block.
2022-05-22 05:24:22.030+00:00 | nioEventLoopGroup-3-10 | INFO  | FastSyncDownloader | Starting fast sync.
2022-05-22 05:24:22.030+00:00 | nioEventLoopGroup-3-10 | INFO  | FastSyncDownloader | Clearing bonsai flat account db
2022-05-22 05:24:22.201+00:00 | nioEventLoopGroup-3-10 | INFO  | PivotSelectorFromPeers | Selecting block number 14821561 as fast sync pivot block.
2022-05-22 05:24:22.202+00:00 | nioEventLoopGroup-3-10 | INFO  | PivotBlockRetriever | Retrieve a pivot block that can be confirmed by at least 5 peers.
2022-05-22 05:24:22.202+00:00 | nioEventLoopGroup-3-10 | INFO  | PivotBlockConfirmer | Confirm pivot block 14821561 with at least 5 peers.
2022-05-22 05:24:22.208+00:00 | nioEventLoopGroup-3-9 | INFO  | PivotBlockConfirmer | Received 1 confirmation(s) for pivot block header 14821561: 0x12090d8b37f555372481b4377d22e16abd7cfa8682597b26e66505efe6c6a796
2022-05-22 05:24:22.285+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Received 2 confirmation(s) for pivot block header 14821561: 0x12090d8b37f555372481b4377d22e16abd7cfa8682597b26e66505efe6c6a796
2022-05-22 05:24:22.335+00:00 | nioEventLoopGroup-3-5 | INFO  | PivotBlockConfirmer | Received 3 confirmation(s) for pivot block header 14821561: 0x12090d8b37f555372481b4377d22e16abd7cfa8682597b26e66505efe6c6a796
2022-05-22 05:24:22.335+00:00 | nioEventLoopGroup-3-2 | INFO  | PivotBlockConfirmer | Received 4 confirmation(s) for pivot block header 14821561: 0x12090d8b37f555372481b4377d22e16abd7cfa8682597b26e66505efe6c6a796
2022-05-22 05:24:24.569+00:00 | nioEventLoopGroup-3-4 | INFO  | PivotBlockConfirmer | Confirmed pivot block at 14821561: 0x12090d8b37f555372481b4377d22e16abd7cfa8682597b26e66505efe6c6a796
2022-05-22 19:32:28.809+00:00 | BesuCommand-Shutdown-Hook | INFO  | FastSyncDownloader | Stopping fast sync

mcdee on May 25, 2022