reth: Unable to sync base-mainnet with op-reth || op-node hangs whilst op-reth is 'pruning'

Describe the bug

Trying to sync op-reth on base-mainnet. Using op-node, op-node often hangs when op-reth is ‘pruning’ (despite this being an archive node sync).

op-node shows following:

INFO [04-07|09:44:54.169] inserted block                           hash=66aadb..89cc2b number=58586 state_root=9b13d9..8889ac timestamp=1,686,906,519 parent=481ef4..e20181 prev_randao=0f039b..bc8f94 fee_recipient=0x4200000000000000000000000000000000000
011 txs=1 update_safe=true
INFO [04-07|09:44:54.170] Sync progress                            reason="processed safe block derived from L1" l2_finalized=df6d3d..9e8c4e:48857 l2_safe=66aadb..89cc2b:58586 l2_pending_safe=66aadb..89cc2b:58586 l2_unsafe=66aadb..89cc2b:58586 l2_backu
p_unsafe=000000..000000:0 l2_time=1,686,906,519 l1_derived=4d7b5c..499d92:17491420
INFO [04-07|09:44:54.170] Found next batch                         batch_type=SingularBatch batch_timestamp=1,686,906,521 parent_hash=66aadb..89cc2b batch_epoch=70d79f..43110c:17491408 txs=0
INFO [04-07|09:44:54.170] generated attributes in payload queue    txs=1 timestamp=1,686,906,521
INFO [04-07|09:44:54.178] inserted block                           hash=3a4ede..5c6eb7 number=58587 state_root=7cfbc1..657780 timestamp=1,686,906,521 parent=66aadb..89cc2b prev_randao=0f039b..bc8f94 fee_recipient=0x4200000000000000000000000000000000000
011 txs=1 update_safe=true
INFO [04-07|09:44:54.178] Sync progress                            reason="processed safe block derived from L1" l2_finalized=df6d3d..9e8c4e:48857 l2_safe=3a4ede..5c6eb7:58587 l2_pending_safe=3a4ede..5c6eb7:58587 l2_unsafe=3a4ede..5c6eb7:58587 l2_backu
p_unsafe=000000..000000:0 l2_time=1,686,906,521 l1_derived=4d7b5c..499d92:17491420
INFO [04-07|09:44:54.178] Found next batch                         batch_type=SingularBatch batch_timestamp=1,686,906,523 parent_hash=3a4ede..5c6eb7 batch_epoch=70d79f..43110c:17491408 txs=0
INFO [04-07|09:44:54.179] generated attributes in payload queue    txs=1 timestamp=1,686,906,523
ERROR[04-07|09:44:54.184] cancelling old block sealing job         payload=0x14ddafb523b060c7
WARN [04-07|09:44:54.186] could not process payload derived from L1 data, dropping batch err="failed to complete building on top of L2 chain 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7:58587, id: 0x14ddafb523b060c7, error (3): up
dated forkchoice, but node is syncing"
ERROR[04-07|09:44:54.186] deposit only block was invalid           parent=3a4ede..5c6eb7:58587 err="failed to complete building on top of L2 chain 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7:58587, id: 0x14ddafb523b060c7, error (
3): updated forkchoice, but node is syncing"
ERROR[04-07|09:44:54.186] Derivation process critical error        err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc7
33a2a9f84f5c6eb7:58587, id: 0x14ddafb523b060c7, error (3): updated forkchoice, but node is syncing"
INFO [04-07|09:44:54.186] State loop returned

N.B. setting block_interval = 5 under the prune setting in reth.toml to something obscene like 5000000000 seems to reduce the frequency of the error. When set at 5 I will see the error every 30 seconds or so, and have to restart op-node. With it set at 5000000000 I’ve not seen the error after ~30 mins of testing.

N.N.B. even setting log level to -vvvv in op-reth made it take much longer for the error to appear in op-node. Maybe 5 mins instead of every 30 seconds.

Steps to reproduce

start op-node ./bin/op-node --network="base-mainnet" --l1=http://127.0.0.1:8545 --l1.beacon=http://127.0.0.1:5052 --l2=http://localhost:9551 --l2.jwt-secret=/data/mev-geth/secret.jwt --rpc.addr=0.0.0.0 --rpc.port=9545

start op-reth ./target/release/op-reth node --chain base --rollup.sequencer-http https://sequencer.base.org --http --ws --authrpc.port 9551 --authrpc.jwtsecret /data/mev-geth/secret.jwt --port 30333 --discovery.port 30333 --ws.port 18546 --http.port 18545 --datadir /data/op-reth

Node logs

2024-04-07T09:44:54.178032Z DEBUG consensus::engine: Canonicalized new head hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 number=58587
2024-04-07T09:44:54.178253Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.178256Z  INFO reth_node_core::events::node: Canonical chain committed number=58587 hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 elapsed=1.403013ms
2024-04-07T09:44:54.178510Z DEBUG payload_builder: building new payload id=0x55e9d3849224cfdf parent_hash=0x0eaeddce536a5301cc5369efa7db499143af1568d961fae94195e3ceb902ae6c parent_number=58538
2024-04-07T09:44:54.178706Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58539
2024-04-07T09:44:54.179034Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xd2b64a930e12aec0cde6fef6ecfa24f21bd73cbac954f0eed98031aae385e117, header: Header { parent_hash: 0x0eaeddce536a5301cc5369efa7db499143af1568d961fae94195e3ceb902ae6c, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x7b5395807646415057b4b085ff51741a3b261695c995a29268d2a32e4bfb579e, transactions_root: 0xa8cc9befd38c48d3d9f88b9bbfd001748a25ea8c87cc48a1c501b51b017d51d4, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58539, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906425, mix_hash: 0x2e6b97ce5835ca7e380b8c6733bc6e0cf2c60531890cc83078eed19028eb389b, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x4f5a1a203b52be0944a69b3b90450dfac20dffe991c2d7f4f67d32d2ca729240, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc1bfc19c8c012764e34c8b9b346eca1153364114c97561d3d6809690e6e744d4, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c800000000000000000000000000000000000000000000000000000000648c25eb0000000000000000000000000000000000000000000000000000000362225146eae384e874a4720dadfabbb8792958faf7cfb7995718da1b5b52d55426ff9c1b00000000000000000000000000000000000000000000000000000000000000030000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.179235Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.179725Z  INFO blockchain_tree: Block is already canonical, ignoring. block_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7
2024-04-07T09:44:54.179918Z DEBUG consensus::engine: [Optimism] Allowing beacon reorg to old head fcu_head_num=58587 current_head_num=58587
2024-04-07T09:44:54.179995Z  INFO reth_node_core::events::node: Canonical chain committed number=58587 hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 elapsed=550.815µs
2024-04-07T09:44:54.180002Z DEBUG pruner: Minimum pruning interval reached previous_tip_block_number=Some(58582) tip_block_number=58587
2024-04-07T09:44:54.180036Z DEBUG pruner: Pruner started tip_block_number=58587
2024-04-07T09:44:54.180041Z  INFO reth_node_core::events::node: Pruner started tip_block_number=58587
2024-04-07T09:44:54.180427Z  INFO reth_payload_builder::service: New payload job created id=0x14ddafb523b060c7 parent=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7
2024-04-07T09:44:54.180015Z DEBUG consensus::engine::hooks: Polled next hook hook="Prune" result=PolledHook { name: "Prune", event: Started, db_access_level: ReadWrite }
2024-04-07T09:44:54.181030Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.181068Z DEBUG payload_builder: building new payload id=0x14ddafb523b060c7 parent_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 parent_number=58587
2024-04-07T09:44:54.181511Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329, header: Header { parent_hash: 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x02c620a6df954aa2fb456d7abb7fba3fdbdea2269a05efeb560091ea8957ea95, transactions_root: 0x5ae1634b08b4f90478bee4ddf84f37ec6069a6511fae22c1a80d7ee69c157ba2, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58588, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906523, mix_hash: 0x0f039b421747a6c6132b14b9178a1fc2c7633f17e974ed3ec36a958e35bc8f94, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x3ed7acd44814f15321c310098d5d2bf288084eade5b78f281ecdcea2141cfd71, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc4d9bf18c0a7b02f2cb6ef25fbb70f0f99e07dba9f36825b0844c11a1fd81cf2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5d000000000000000000000000000000000000000000000000000000000648c264b0000000000000000000000000000000000000000000000000000000358e19b7270d79f07b4b2d8db4a3a2dc3ad1297d988c6f8bed5059556c9ef26d42043110c00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.181538Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.181588Z DEBUG payload_builder: building new payload id=0x14ddafb523b060c7 parent_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 parent_number=58587
2024-04-07T09:44:54.181955Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329, header: Header { parent_hash: 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x02c620a6df954aa2fb456d7abb7fba3fdbdea2269a05efeb560091ea8957ea95, transactions_root: 0x5ae1634b08b4f90478bee4ddf84f37ec6069a6511fae22c1a80d7ee69c157ba2, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58588, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906523, mix_hash: 0x0f039b421747a6c6132b14b9178a1fc2c7633f17e974ed3ec36a958e35bc8f94, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x3ed7acd44814f15321c310098d5d2bf288084eade5b78f281ecdcea2141cfd71, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc4d9bf18c0a7b02f2cb6ef25fbb70f0f99e07dba9f36825b0844c11a1fd81cf2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5d000000000000000000000000000000000000000000000000000000000648c264b0000000000000000000000000000000000000000000000000000000358e19b7270d79f07b4b2d8db4a3a2dc3ad1297d988c6f8bed5059556c9ef26d42043110c00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.181980Z DEBUG payload_builder: resolving fallback payload as best payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.182291Z DEBUG payload_builder: building new payload id=0x1afa30385c2a133b parent_hash=0x42d573473e2e53140ed1335ef660ad403179c99ac4cf8142b04c03c00be4dce6 parent_number=58201
2024-04-07T09:44:54.182404Z DEBUG payload_builder: building new payload id=0x9eb78b76c133d825 parent_hash=0x3e97904bbd92511c38bc3597c4c9aa0e4c5707c30f92404d21ab666cdf889898 parent_number=58487
2024-04-07T09:44:54.182806Z DEBUG payload_builder: resolving better payload
2024-04-07T09:44:54.183649Z DEBUG blockchain_tree: Appending block to canonical chain head=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 parent=(58587, 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7)
2024-04-07T09:44:54.183863Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=2
2024-04-07T09:44:54.184265Z DEBUG blockchain_tree::chain: Validated state root number=58588 hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 elapsed=438.048µs
2024-04-07T09:44:54.184308Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.184323Z  INFO reth_node_core::events::node: Block added to canonical chain number=58588 hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 peers=0 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=976.4µs
2024-04-07T09:44:54.184681Z  WARN consensus::engine: Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator. hook=Prune head_block_hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 safe_block_hash=0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329 finalized_block_hash=0xdf6d3d5c2496182bf82f59ea7fa1ce75f8f4012e1c2aa44b9b9b3d4e859e8c4e
2024-04-07T09:44:54.184695Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.185550Z DEBUG payload_builder: no best payload yet to resolve, building empty payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.185597Z DEBUG payload_builder: building new payload id=0x14ddafb523b060c7 parent_hash=0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7 parent_number=58587
2024-04-07T09:44:54.186031Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xf53567fd67bd78b41a3d9c0bd1e73f29a1045c4936984b01d33d0b7144843329, header: Header { parent_hash: 0x3a4ede2caecee90621cfb23cfb25a03e70129aca7944fbc733a2a9f84f5c6eb7, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x02c620a6df954aa2fb456d7abb7fba3fdbdea2269a05efeb560091ea8957ea95, transactions_root: 0x5ae1634b08b4f90478bee4ddf84f37ec6069a6511fae22c1a80d7ee69c157ba2, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58588, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906523, mix_hash: 0x0f039b421747a6c6132b14b9178a1fc2c7633f17e974ed3ec36a958e35bc8f94, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x3ed7acd44814f15321c310098d5d2bf288084eade5b78f281ecdcea2141cfd71, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc4d9bf18c0a7b02f2cb6ef25fbb70f0f99e07dba9f36825b0844c11a1fd81cf2, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5d000000000000000000000000000000000000000000000000000000000648c264b0000000000000000000000000000000000000000000000000000000358e19b7270d79f07b4b2d8db4a3a2dc3ad1297d988c6f8bed5059556c9ef26d42043110c00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.186073Z DEBUG payload_builder: resolving fallback payload as best payload id=0x14ddafb523b060c7
2024-04-07T09:44:54.186091Z DEBUG payload_builder: resolving best payload
2024-04-07T09:44:54.188357Z DEBUG payload_builder: building new payload id=0xba9b204a7803fbd3 parent_hash=0xd2b64a930e12aec0cde6fef6ecfa24f21bd73cbac954f0eed98031aae385e117 parent_number=58539
2024-04-07T09:44:54.188503Z DEBUG payload_builder: building new payload id=0xf215c90466ac9926 parent_hash=0x87b6beab35817901b2dc15f0686a1864bf8a0450de5e949d4033a5f951d3109e parent_number=58488
2024-04-07T09:44:54.188576Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58540
2024-04-07T09:44:54.188921Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xa4165e67354c3022630d96d8cf3fa31190c0aaa68869089dff428bc9b78444a4, header: Header { parent_hash: 0xd2b64a930e12aec0cde6fef6ecfa24f21bd73cbac954f0eed98031aae385e117, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6df72d55185e73207564fbbf2ec515af8971d8f14d35bdb77b778b7ab50d1722, transactions_root: 0xb20b44f5397c71bb924858e586c47de1b6aa4a21c7878aa8d875f3adc3108f27, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58540, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906427, mix_hash: 0x2e6b97ce5835ca7e380b8c6733bc6e0cf2c60531890cc83078eed19028eb389b, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x920ffd8bc230bdfe1f61b3bed7712d067b5f137f03c3f4503029f6e390e4c7d5, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0x75e36bd155036249db3de9f81d84a17c18fca568da2893673c285822a2fad3ce, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c800000000000000000000000000000000000000000000000000000000648c25eb0000000000000000000000000000000000000000000000000000000362225146eae384e874a4720dadfabbb8792958faf7cfb7995718da1b5b52d55426ff9c1b00000000000000000000000000000000000000000000000000000000000000040000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.189091Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.191100Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=48857 prune_mode=Before(48858)
2024-04-07T09:44:54.191146Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=48857 prune_mode=Before(48858) output.pruned=0
2024-04-07T09:44:54.191153Z DEBUG pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=48857 prune_mode=Before(48858)
2024-04-07T09:44:54.191162Z DEBUG pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=48857 prune_mode=Before(48858) output.pruned=0
2024-04-07T09:44:54.191169Z DEBUG pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=48857 prune_mode=Before(48858)
2024-04-07T09:44:54.191175Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=48857 prune_mode=Before(48858) output.pruned=0
2024-04-07T09:44:54.191205Z DEBUG storage::db::mdbx: Commit total_duration=18.763µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-07T09:44:54.191231Z DEBUG pruner: Pruner finished tip_block_number=58587 elapsed=11.179791ms delete_limit=8500 progress=Finished stats={}
2024-04-07T09:44:54.191270Z  INFO reth_node_core::events::node: Pruner finished tip_block_number=58587 elapsed=11.179791ms stats={}
2024-04-07T09:44:54.191282Z DEBUG consensus::engine::hooks: Polled running hook with db write access hook="Prune" result=PolledHook { name: "Prune", event: Finished(Ok(())), db_access_level: ReadWrite }
2024-04-07T09:44:54.191468Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-07T09:44:54.193700Z DEBUG payload_builder: building new payload id=0x1a524539692c8fa0 parent_hash=0xba8e557c674771dec883112af0e9e230a5aeb2ca3e5e815eb2e5a73ce824cc39 parent_number=58202
2024-04-07T09:44:54.195250Z DEBUG payload_builder: building new payload id=0xbe169363e05f7bb1 parent_hash=0xdc1b728ec2cac433e3423d2c9c48ed5e81ff597bde5f80d51914cfffe883a903 parent_number=58258
2024-04-07T09:44:54.196768Z DEBUG payload_builder: building new payload id=0x1d7ec23d848a530c parent_hash=0xea290dda7668ccd114aa6e5099ef8d43c1dd6ccbc88a5c2b0aca6301cdbf1b74 parent_number=58489
2024-04-07T09:44:54.198178Z DEBUG payload_builder: building new payload id=0x260a466b18dc9663 parent_hash=0x6975478d2b5c2fb6a658697cccd5a537b476fc66b434c2e5f19744f7e4aaa1f3 parent_number=58259
2024-04-07T09:44:54.198328Z DEBUG payload_builder: building new payload id=0x73ffe70330eb135b parent_hash=0xa4165e67354c3022630d96d8cf3fa31190c0aaa68869089dff428bc9b78444a4 parent_number=58540
2024-04-07T09:44:54.198472Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58541
2024-04-07T09:44:54.198804Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x9a1637cd5266813e2c1552652693b6e62e6a55ce92fcbceef7b909f3c7e53871, header: Header { parent_hash: 0xa4165e67354c3022630d96d8cf3fa31190c0aaa68869089dff428bc9b78444a4, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x8cf98309b43cfca46d63f10844fc6e1dd864693f769721b71c9ef9025e4ce03b, transactions_root: 0x430eb5394508edbb1f2f73928a5c6120f2cfce0b9a650704405c761f62c97c89, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58541, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906429, mix_hash: 0x2e6b97ce5835ca7e380b8c6733bc6e0cf2c60531890cc83078eed19028eb389b, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xc766d7dd3d1a8089d03f2e3649e5b5b5f124053cded78d41a42d749b3eea2e04, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xfb486b24abc723b921ba6a61d15edd59e06ae27a185a8b29c59c9445291623fe, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c800000000000000000000000000000000000000000000000000000000648c25eb0000000000000000000000000000000000000000000000000000000362225146eae384e874a4720dadfabbb8792958faf7cfb7995718da1b5b52d55426ff9c1b00000000000000000000000000000000000000000000000000000000000000050000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.199055Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.204277Z DEBUG payload_builder: building new payload id=0x4bba997a7c12b9e5 parent_hash=0x4bc58cc28e8b6c0746dfc99a62472c7e6c3089b8fcb1deac2c545f139fbb96ca parent_number=58490
2024-04-07T09:44:54.205958Z DEBUG payload_builder: building new payload id=0x369b307a0a932359 parent_hash=0x5735beadd4944533f6e167419f86a8bec195d9316e4a1b92021e2c195383410b parent_number=58260
2024-04-07T09:44:54.208306Z DEBUG payload_builder: building new payload id=0x14ec4c52c4c4d2ee parent_hash=0x06922fe25a277ae3c01b9effaeefcb8d163bcf4017bd36b9859c20147950f3b2 parent_number=58203
2024-04-07T09:44:54.208339Z DEBUG payload_builder: building new payload id=0x7011b34027e78fdb parent_hash=0x7776dedeef4d550e8e1f4b65d51b547b2349ebe03697eb0f3f2bf116efa74039 parent_number=58261
2024-04-07T09:44:54.208428Z DEBUG payload_builder: building new payload id=0xb2860c84d5470f0c parent_hash=0x9a1637cd5266813e2c1552652693b6e62e6a55ce92fcbceef7b909f3c7e53871 parent_number=58541
2024-04-07T09:44:54.208581Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58542
2024-04-07T09:44:54.208943Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0xcd6385f63a0443e3fd57e101332ecb88e11a18a70013467cc55b889fced40966, header: Header { parent_hash: 0x9a1637cd5266813e2c1552652693b6e62e6a55ce92fcbceef7b909f3c7e53871, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x8ceb5ec4c4b080e39437dc61191c35acfbe1c1dc16d41155a423faa1e952d919, transactions_root: 0x35519e7c1d2ed2678c7c4fd0686a9874640d0bce0174e1814dc055ba9f6c5df8, receipts_root: 0x0e7c255df3e2b7ca4d55b82047531f7f3e7437ab1eb33e22eca70653c874982d, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58542, gas_limit: 30000000, gas_used: 50501, timestamp: 1686906431, mix_hash: 0x49ccf354359ba9aa65866d292b5f461c54a3605a70b06db21687afd260154706, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0xa4d3f6bd3f2be0e300390658d41a6ffc340ee6f7ed92ee32c8c9e8da85286500, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xa30069c0f9b2a074d348962442cea4bc1132c17ef666d5ed0896e17efe3acd97, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c900000000000000000000000000000000000000000000000000000000648c25f700000000000000000000000000000000000000000000000000000003432d7dbdd10eaf160ab24585f4b9e061e958ecd2d567a9b34faa0793d4ed740382f485db00000000000000000000000000000000000000000000000000000000000000000000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.209097Z DEBUG payload_builder: building new payload id=0x0d14f4c100545f9d parent_hash=0x9c18dae0ac7e83b893ee489aae89acd1d01e52b463d3350bde3b7c279d1c4300 parent_number=58491
2024-04-07T09:44:54.209237Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.213710Z DEBUG payload_builder: building new payload id=0x441c548376a6baef parent_hash=0xaa2d17125ec3088f5f856ac718e3bc622d6a5df1f52138ae671104cf716efb92 parent_number=58262
2024-04-07T09:44:54.215146Z DEBUG payload_builder: building new payload id=0xa97a415220c9e159 parent_hash=0x193e5c2c9f3e8650616d73b80b773b376978d74de1830e946f2fd89e6c03eae2 parent_number=58492
2024-04-07T09:44:54.216510Z DEBUG payload_builder: building new payload id=0x750e432b897393ae parent_hash=0x87331051e41c45d6c86fb7a053a43b352c6e0b2b833259aa931a8b44e8de71c1 parent_number=58263
2024-04-07T09:44:54.219919Z DEBUG payload_builder: building new payload id=0x7d18e3dbf094bd23 parent_hash=0xcd6385f63a0443e3fd57e101332ecb88e11a18a70013467cc55b889fced40966 parent_number=58542
2024-04-07T09:44:54.220145Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58543
2024-04-07T09:44:54.220201Z DEBUG payload_builder: building new payload id=0x6f18699b1180833c parent_hash=0x04deb25b5a95d78e460b0e13429ff94c396ced02396777b7b279f4e57b1bd5d6 parent_number=58406
2024-04-07T09:44:54.220496Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x72fc8d58f91fd3f09b5091a5dd4121873eb46a49808dbf71aeb848ab91e1163b, header: Header { parent_hash: 0xcd6385f63a0443e3fd57e101332ecb88e11a18a70013467cc55b889fced40966, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x8b546fe63827af69faf63a53d4475652ff585db568756b93fc91097b69cc469b, transactions_root: 0xc3f81934a3e121d72ae7da996d14a847efc01c5c23fdfabce9c0c563c6ac2c2a, receipts_root: 0x637d9e6847368bb52b56e90dd608facc34122c153872e8733cb0b01caa581f85, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58543, gas_limit: 30000000, gas_used: 64013, timestamp: 1686906433, mix_hash: 0x49ccf354359ba9aa65866d292b5f461c54a3605a70b06db21687afd260154706, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x50272fb4bc55ff1b6df13040eddaaea8d1a580243ee736250429ffec37e350bd, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xc43bf4a3e7c9777891d52297f2eda3b81cfabfeb0094a109b410aec9deb60e87, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c900000000000000000000000000000000000000000000000000000000648c25f700000000000000000000000000000000000000000000000000000003432d7dbdd10eaf160ab24585f4b9e061e958ecd2d567a9b34faa0793d4ed740382f485db00000000000000000000000000000000000000000000000000000000000000010000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.220641Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.222903Z DEBUG payload_builder: building new payload id=0x73e5f84fac2f7ce2 parent_hash=0xf9fd5d17664fb8a936483f069b58859884d7612a4b9e2927fd14742230644fbc parent_number=58493
2024-04-07T09:44:54.224437Z DEBUG payload_builder: building new payload id=0x71b84955ebbda7f0 parent_hash=0xf2a28233d35e569739d1e61d5cf928264e4a8f62a6b71dd43e6c0418015e003b parent_number=58264
2024-04-07T09:44:54.226899Z DEBUG payload_builder: building new payload id=0x9d5c805739102ce9 parent_hash=0xe3a372e7583af9122492e4d93aeaf5198f34daf9e3828d765e23459a576ef677 parent_number=58204
2024-04-07T09:44:54.226899Z DEBUG payload_builder: building new payload id=0x88922a230887bb78 parent_hash=0x7b213573e3d35904660632fb79c2afd4fc105c72e519f2d874da0444f9616b3c parent_number=58407
2024-04-07T09:44:54.227091Z DEBUG payload_builder: building new payload id=0x03f10b6d36148495 parent_hash=0x7da5fb624f431f93708d6364237da2276bb595c18eaec525c3a2987489ec656b parent_number=58265
2024-04-07T09:44:54.227112Z DEBUG payload_builder: building new payload id=0xb4dda4e18a04abe7 parent_hash=0x72fc8d58f91fd3f09b5091a5dd4121873eb46a49808dbf71aeb848ab91e1163b parent_number=58543
2024-04-07T09:44:54.227252Z  WARN provider::historical_sp: Attempt to calculate state root for an old block might result in OOM, tread carefully target=58544
2024-04-07T09:44:54.227595Z DEBUG payload_builder: sealed built block sealed_block=SealedBlock { header: SealedHeader { hash: 0x87c160a636c53b96574226dbf8a673a99bc9e73f30848f656143ea5cdae7694c, header: Header { parent_hash: 0x72fc8d58f91fd3f09b5091a5dd4121873eb46a49808dbf71aeb848ab91e1163b, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0xad50be7c814f8079fd6b230169788c7566907674716e5685e1839bbb9b0d1c7d, transactions_root: 0x99090cac9b4e48e0bbdf46120fde5319a8c6e7866126eddcf6fbfba090457b48, receipts_root: 0x3f71ebd8822765e8146170b0f930f55f753dbb867f55e4e605894016cd32f1e2, withdrawals_root: None, logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0x0_U256, number: 58544, gas_limit: 30000000, gas_used: 46913, timestamp: 1686906435, mix_hash: 0x49ccf354359ba9aa65866d292b5f461c54a3605a70b06db21687afd260154706, nonce: 0, base_fee_per_gas: Some(50), blob_gas_used: None, excess_blob_gas: None, parent_beacon_block_root: None, extra_data: 0x } }, body: [TransactionSigned { hash: 0x566c6b54ee1a6a912fbc0310b2723a01ba27e37d2a52cf8c725cbf17ff1f8fb4, signature: Signature { r: 0x0_U256, s: 0x0_U256, odd_y_parity: false }, transaction: Deposit(TxDeposit { source_hash: 0xb4cc0e07ec6f39695e04d11a8da90786fa34b8925251429502d2e88ec73c2621, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0x0_U256, gas_limit: 1000000, is_system_transaction: false, input: 0x015d8eb900000000000000000000000000000000000000000000000000000000010ae5c900000000000000000000000000000000000000000000000000000000648c25f700000000000000000000000000000000000000000000000000000003432d7dbdd10eaf160ab24585f4b9e061e958ecd2d567a9b34faa0793d4ed740382f485db00000000000000000000000000000000000000000000000000000000000000020000000000000000000000005050f69a9786f081509234f1a7f4684b5e5b76c900000000000000000000000000000000000000000000000000000000000000bc00000000000000000000000000000000000000000000000000000000000a6fe0 }) }], ommers: [], withdrawals: None }
2024-04-07T09:44:54.227775Z DEBUG payload_builder: built better payload value=0
2024-04-07T09:44:54.232022Z DEBUG payload_builder: building new payload id=0xe2075cf95b7dec75 parent_hash=0x22e0b6bcccecb0dfb863482ffcbddbe9884a0075e2847cbd3c9fcf9c440822ae parent_number=58408
2024-04-07T09:44:54.233551Z DEBUG payload_builder: building new payload id=0x73493bfdb8915fa1 parent_hash=0x0ab4185c8fbf71734efd995f585a1bb5052647f3b3baeb805fa0c7a79796fef2 parent_number=58266
2024-04-07T09:44:54.238005Z DEBUG payload_builder: building new payload id=0x918ff1adaa7a89bc parent_hash=0xcf1148ff3427f8c1c1e33a373b21596eb5c46fced2154e16d2efeb31c7505725 parent_number=58409
......
......
......
2024-04-07T09:44:54.366736Z DEBUG payload_builder: building new payload id=0x013e88ed859f72ad parent_hash=0xb4e82429cf20ef15ea60b159b7ffaff8b85de1614f9e111411d8673421f063c9 parent_number=58220
2024-04-07T09:44:54.369550Z DEBUG payload_builder: building new payload id=0x69c641703ff6b29e parent_hash=0xd67a6094661ca2052f71247655bf4e7f8c53bf1e16576f0b3366eb2404e804ce parent_number=58350
2024-04-07T09:44:54.371029Z DEBUG payload_builder: building new payload id=0x087fbff9a979b7fd parent_hash=0x96f3e254da22c7131537b8810bd0094d244e69374798d5fa1d9ad47613a85e7e parent_number=58221
2024-04-07T09:44:54.371276Z DEBUG payload_builder: building new payload id=0xa35c374ff84ffa5d parent_hash=0x62b37e19857462686b74df0f795072cad3dc761598669b33f0b49d3dfdda8302 parent_number=58351
2024-04-07T09:44:54.373679Z DEBUG payload_builder: building new payload id=0x4071fe2e79d61fac parent_hash=0xa8989cdd042c3268f00dbac3a65f5e6c6046e79d07f93d344ac906cf03e10842 parent_number=58222
2024-04-07T09:44:54.378480Z DEBUG payload_builder: building new payload id=0x26eaf4207312fe83 parent_hash=0x91d0f29a54be9358809160531d98de9c2d3f81afcc7c7fcc7fb605735ef5a32c parent_number=58223
2024-04-07T09:44:54.378480Z DEBUG payload_builder: building new payload id=0xcb56b34e39b1423b parent_hash=0x2fdf5627712726df13271293b1f1457f015ba8e3bcaf2443662992eef4fcafd2 parent_number=58352

Platform(s)

Linux (x86)

What version/commit are you on?

./target/release/op-reth --version reth Version: 0.2.0-beta.4 Commit SHA: c04dbe6e9 Build Timestamp: 2024-04-07T07:19:12.388844682Z Build Features: jemalloc,optimism Build Profile: release

What database version are you on?

NA

What type of node are you running?

Archive (default)

What prune config do you use, if any?

[prune] block_interval = 5

If you’ve built Reth from source, provide the full command you used

No response

Code of Conduct

  • I agree to follow the Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 3 months ago
  • Reactions: 1
  • Comments: 18 (4 by maintainers)

Most upvoted comments

[prune] block_interval = 999999999 in reth.toml also ‘resolves’

@0xZerohero thanks, good to know syncing is smooth with disabled pruner. archive full nodes prune history. atm timeout is only soft for history pruning (is only checked when history pruning returns). so atm timeout only helps on full archive nodes, not archive full nodes.

I can verify this still happens on latest HEAD 711de3b08d62857d2c3959468eb327c443f776ca

Logs op-node:

parent_hash=0xcc271f0a219e863a1383e1dddaff54d80574b2a045bb8040ee79cd0fd43432fa batch_epoch=0xe8ca726a1a92fcb6822526536096dc55de2670ac56606f1921d42a48e465472b:17792388 txs=0 parent=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920:1884073 parent_time=1690557493
t=2024-04-16T10:49:37+0200 lvl=info msg="Generating next batch" epoch=0x9e1d2ebfe5f0aaa4903655776c433b91ef7bb9a019aa2dc4a5d33f57b3fbe364:17792394 timestamp=1690557495
t=2024-04-16T10:49:37+0200 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1690557495
t=2024-04-16T10:49:37+0200 lvl=info msg="inserted block" hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 number=1884074 state_root=0x924daa236c26ae48767a1375f4f76db828e3f0babbe0a2849c869a17b190d6d9 timestamp=1690557495 parent=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 prev_randao=0x75be4217e571ba7459592cc119a099c98f9e79b595b08c3701e5e60e766ab338 fee_recipient=0x4200000000000000000000000000000000000011 txs=1 update_safe=true
t=2024-04-16T10:49:37+0200 lvl=info msg="Sync progress" reason="processed safe block derived from L1" l2_finalized=0xb9fa1d4a1d109d025f749ae7e0f129f5a2836e40f155c7d8b735367d04c99834:1883777 l2_safe=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 l2_pending_safe=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 l2_unsafe=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1690557495 l1_derived=0x6d449d294c8950471694e7b9cf2c2f3443ab76d27a5286ad8cd5a9113df6e3b0:17795994
t=2024-04-16T10:49:37+0200 lvl=warn msg="ignoring batch with mismatching parent hash" batch_index=0 batch_type=SingularBatch batch_timestamp=1690557497 parent_hash=0xa89bf2fa0f65975a9265cfc107b5807e3d4b9b241b2b72775b10dbd9168659c1 batch_epoch=0xe8ca726a1a92fcb6822526536096dc55de2670ac56606f1921d42a48e465472b:17792388 txs=0 current_safe_head=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22
t=2024-04-16T10:49:37+0200 lvl=warn msg="Dropping batch" batch_type=SingularBatch batch_timestamp=1690557497 parent_hash=0xa89bf2fa0f65975a9265cfc107b5807e3d4b9b241b2b72775b10dbd9168659c1 batch_epoch=0xe8ca726a1a92fcb6822526536096dc55de2670ac56606f1921d42a48e465472b:17792388 txs=0 parent=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 parent_time=1690557495
t=2024-04-16T10:49:37+0200 lvl=info msg="Generating next batch" epoch=0x9e1d2ebfe5f0aaa4903655776c433b91ef7bb9a019aa2dc4a5d33f57b3fbe364:17792394 timestamp=1690557497
t=2024-04-16T10:49:37+0200 lvl=info msg="generated attributes in payload queue" txs=1 timestamp=1690557497
t=2024-04-16T10:49:37+0200 lvl=error msg="cancelling old block sealing job" payload=0x2b2c84d225fe8a46
t=2024-04-16T10:49:37+0200 lvl=warn msg="Failed to get payload" payload_id=0x2b2c84d225fe8a46 payload_id=0x2b2c84d225fe8a46 err="Unknown payload"
t=2024-04-16T10:49:37+0200 lvl=error msg="failed to cancel block building job" payload=0x2b2c84d225fe8a46 err="unrecognized rpc error: Unknown payload"
t=2024-04-16T10:49:37+0200 lvl=warn msg="could not process payload derived from L1 data, dropping batch" err="failed to complete building on top of L2 chain 0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074, id: 0x2b2c84d225fe8a46, error (3): updated forkchoice, but node is syncing"
t=2024-04-16T10:49:37+0200 lvl=error msg="deposit only block was invalid" parent=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074 err="failed to complete building on top of L2 chain 0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074, id: 0x2b2c84d225fe8a46, error (3): updated forkchoice, but node is syncing"
t=2024-04-16T10:49:37+0200 lvl=error msg="Derivation process critical error" err="engine stage failed: crit: failed to process block with only deposit transactions: failed to complete building on top of L2 chain 0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22:1884074, id: 0x2b2c84d225fe8a46, error (3): updated forkchoice, but node is syncing"
t=2024-04-16T10:49:37+0200 lvl=info msg="State loop returned"

Logs reth:

2024-04-16T08:49:37.252495Z  INFO Block added to canonical chain number=1884073 hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 peers=3 txs=1 mgas=0.064 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=478.423µs
2024-04-16T08:49:37.253398Z  INFO Canonical chain committed number=1884073 hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 elapsed=731.866µs
2024-04-16T08:49:37.303133Z  INFO Block is already canonical, ignoring. block_hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920
2024-04-16T08:49:37.303168Z  INFO Canonical chain committed number=1884073 hash=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920 elapsed=46.607µs
2024-04-16T08:49:37.303210Z  INFO New payload job created id=0x8b627a82aa609590 parent=0xcf9fcffb317b4d3eb1392ec64c4f6f4b8e58733833ac8499124dc47698fdc920
2024-04-16T08:49:37.304555Z  INFO Block added to canonical chain number=1884074 hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 peers=3 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=539.216µs
2024-04-16T08:49:37.305426Z  INFO Canonical chain committed number=1884074 hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 elapsed=761.801µs
2024-04-16T08:49:37.371254Z  INFO Block is already canonical, ignoring. block_hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22
2024-04-16T08:49:37.371586Z  INFO Pruner started tip_block_number=1884074
2024-04-16T08:49:37.371602Z  INFO Canonical chain committed number=1884074 hash=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22 elapsed=51.606µs
2024-04-16T08:49:37.371678Z  INFO New payload job created id=0x2b2c84d225fe8a46 parent=0x485e30165368d9d1ff36b21f1c176cc12d1e0ba97f6b1fb0667cfc673b9d0d22
2024-04-16T08:49:37.374554Z  INFO Block added to canonical chain number=1884075 hash=0xbd3b7e03cf9e9e5cfaf0c6fd58160032b24baecc6051f9359a1794005762860d peers=3 txs=1 mgas=0.047 full=0.2% base_fee=0.00gwei blobs=0 excess_blobs=0 elapsed=1.231837ms
2024-04-16T08:49:37.374999Z  WARN Hook is in progress, skipping forkchoice update. This may affect the performance of your node as a validator. hook=Prune head_block_hash=0xbd3b7e03cf9e9e5cfaf0c6fd58160032b24baecc6051f9359a1794005762860d safe_block_hash=0xbd3b7e03cf9e9e5cfaf0c6fd58160032b24baecc6051f9359a1794005762860d finalized_block_hash=0xb9fa1d4a1d109d025f749ae7e0f129f5a2836e40f155c7d8b735367d04c99834
2024-04-16T08:49:37.377031Z  INFO Pruner finished tip_block_number=1884074 elapsed=5.681868ms stats={}
2024-04-16T08:50:01.192094Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:50:26.192002Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:50:51.191779Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:51:16.191845Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:51:41.191966Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:52:06.191916Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:52:31.192208Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:52:56.191791Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:53:21.191788Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:53:46.191789Z  INFO Status connected_peers=3 freelist=71 latest_block=1884074
2024-04-16T08:54:08.183008Z  WARN Beacon client online, but no consensus updates received for a while. Please fix your beacon client to follow the chain! period=270.808010513s

This occurs running both v0.2.0-beta.5 and head