besu: Bonsai trie initial sync fails with `World State Root does not match expected value` (testing only)

Description

With an initial mainnet fast sync using bonsai tries, the sync fails with this error message:

eth1_1  | 2021-05-05 01:40:17.950+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | CompleteTaskStep | Downloaded 883300000 world state nodes. At least 82 nodes remaining.
eth1_1  | 2021-05-05 01:40:47.085+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | DefaultSynchronizer | Fast sync completed successfully with pivot block 12358887
eth1_1  | 2021-05-05 01:40:47.091+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | FullSyncDownloader | Starting full sync.
eth1_1  | 2021-05-05 01:40:47.099+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | WorldDownloadState | Finished downloading world state from peers
eth1_1  | 2021-05-05 01:40:47.570+00:00 | nioEventLoopGroup-3-1 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x017cb4270271491285... at block 12358887
eth1_1  | 2021-05-05 01:41:05.053+00:00 | EthScheduler-Services-34 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay.
eth1_1  | java.util.concurrent.CompletionException: java.lang.RuntimeException: World State Root does not match expected value, header 0xec071e09803825ae0438af2fd562ef168f7a301df7e4b4867f1108c125a2742d calculated 0xc4374cf503d80e0ec722776b55e7f505c290bd55ef2cbbcf78fc1847a0b691b4
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1093)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2152)
eth1_1  |       at org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:179)
eth1_1  |       at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:158)
eth1_1  |       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
eth1_1  |       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
eth1_1  |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
eth1_1  |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
eth1_1  |       at java.base/java.lang.Thread.run(Thread.java:832)
eth1_1  | Caused by: java.lang.RuntimeException: World State Root does not match expected value, header 0xec071e09803825ae0438af2fd562ef168f7a301df7e4b4867f1108c125a2742d calculated 0xc4374cf503d80e0ec722776b55e7f505c290bd55ef2cbbcf78fc1847a0b691b4

I understand Bonsai tries are not production ready, this sync was done for testing purposes only.

Versions (Add all that apply)

Software version: besu/v21.1.6-dev-425db2a9/linux-x86_64/oracle_openjdk-java-15
Java version: 15
OS Name & Version: Ubuntu 20.04

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (8 by maintainers)

Most upvoted comments

Hi, we modified the fix and did some tests. This new fix is fine and the tests were green. https://github.com/hyperledger/besu/pull/2934

We merged . Feel free to tell us if you still have an issue

matkt on Oct 29, 2021

Thanks for your help, we just added a fix to resolve this issue. Feel free to reopen the ticket if you have the issue again

matkt on Jun 30, 2021

This is the output for the pivot block:

curl --location --request POST 'http://localhost:8547' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["12475948", true], "id":1}'

  "jsonrpc" : "2.0",
  "id" : 1,
  "result" : {
    "number" : "0xbe5e2c",
    "hash" : "0xb7166f29c1b0db4b77a3ef32fe3886bb1724df2f8736ad03a2f78deab8f06638",
    "mixHash" : "0xf12dd642d093f7f46181415d9f40482ce2b6a185176a4e0eb8b6c9fbcceabfeb",
    "parentHash" : "0xcecf4bc8685c3566c0430012ddead18f91014e59e53a6aed4449d7c6de1b2a1a",
    "nonce" : "0xac8c5f6caece7bee",
    "sha3Uncles" : "0xa757be5f27a4611f860bc51e7b414e6e3f706aff0fc3836e3e875cd6a8e1838e",
    "logsBloom" : "0x0220809205020810808a54b0800050809620000010100747a0232430068002e1090c244204800080080248123001210006000000490d20214104280a02e4602430c81c81245000694c34800801302821480210400940160097800e00a90040121084048402020802010610108040484140098a000684066007241192114f1401109821088230a214154008489100080485b00b1101018238684000420818291802120188190021400a8c548880040000028042010022020c00664440010802100610201620800048a4a00480208004bc08408b42844414105a05208210922215083025108820410b00c012850212000040208000014080c41000108810021220",
    "transactionsRoot" : "0x9f1ae146f568f8c1dbc92f1b6bb01982118a90d4f4b2a1db445f7d5687a677ed",
    "stateRoot" : "0x47e9468c63a94d109a7d5fd74e01139c18bcc3baab876346a6e04d6975140332",
    "receiptsRoot" : "0x594bdc3db4d5cab2fa7decfb5ad5d9203e46df71a608f0a6de4cb041ea96f60f",
    "miner" : "0xf20b338752976878754518183873602902360704",
    "difficulty" : "0x1cb711b6732d4f",
    "totalDifficulty" : "0x54c8a5faf92b8c6fecf",
    "extraData" : "0xe4b883e5bda9e7a59ee4bb99e9b1bc0b0321",
    "size" : "0x1114f",
    "gasLimit" : "0xe4e1c0",
    "gasUsed" : "0xe4a4db",
    "timestamp" : "0x60a74b18",
    "uncles" : [ "0x7cf9200a873a7a30198eb69de284a7a3295ce4d09ed6ad6315be139a6c0b778b" ],
    "transactions" : [ {

The state root matches etherscan. Before restart it showed World State Root does not match expected value, header 0xa2792be39009e2b019cd0ed857b7dab92f1607917a37567ce6a576d8ec9d602a calculated 0x8cdcfca1f16f793daf37a921ba7760ef3c3224f66030463b40e7f4fbe8085e3b

I don’t see those hex values anywhere in that block, in its parent, or the uncle.

Where to from here?

It is the new block that Besu try to import after the fastsync. So obviously if we stop the fastsync in the middle the data in database are corrupted and we are not able to validate new blocks. I’ll try to investigate this. Thanks a lot for all this informations 😃

matkt on May 26, 2021