reth: Reth on holesky looks to have a corrupted DB and entered restart loop after rebooting

Describe the bug

Reth was only reporting it was connected to peers and lighthouse becaon said and validators weren’t happy, so I decided to reboot the computer and it went into a restart loop.

Steps to reproduce

I don’t know how to reproduce the problem, but here is what I observed.

This morning I looked at the reth log output from journalctl on my Arch Linux machine and it wasn’t not doing much

Apr 02 10:21:54 ethel reth[44917]: 2024-04-02T17:21:54.560177Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:22:19 ethel reth[44917]: 2024-04-02T17:22:19.560796Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:22:41 ethel reth[44917]: 2024-04-02T17:22:41.514158Z  WARN Beacon client online, but no consensus updates received for a while. Please fix your beacon client to follow the chain! period=42399.460567189s
Apr 02 10:22:44 ethel reth[44917]: 2024-04-02T17:22:44.560510Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:23:09 ethel reth[44917]: 2024-04-02T17:23:09.560047Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:23:34 ethel reth[44917]: 2024-04-02T17:23:34.560511Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:23:59 ethel reth[44917]: 2024-04-02T17:23:59.560034Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:24:24 ethel reth[44917]: 2024-04-02T17:24:24.560481Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352

And the lighthouse beacon and validator nodes weren’t happy, so I restarted them leaving reth running. In retrospect I probably should have left beacon and validator alone and restarted reth.

But anyway, here is beacon restarting:

Apr 02 10:23:36 ethel lighthouse[1511]: Apr 02 17:23:36.002 WARN Error processing HTTP API request       method: POST, path: /eth/v1/validator/duties/attester/42126, status: 400 Bad Request, elapsed: 75.464µs
Apr 02 10:23:42 ethel lighthouse[1511]: Apr 02 17:23:42.000 INFO Syncing                                 est_time: --, distance: 3598 slots (11 hrs 59 mins), peers: 43, service: slot_notifier
Apr 02 10:23:42 ethel lighthouse[1511]: Apr 02 17:23:42.000 WARN Syncing deposit contract block cache    est_blocks_remaining: 2118, service: slot_notifier
Apr 02 10:23:43 ethel lighthouse[1511]: Apr 02 17:23:43.662 WARN Failed to sync chain built on invalid parent, advice: check execution node for corruption then restart it and Lighthouse, parent_root: 0x9f2eaf6bde5a8eb145084739a618c94fb48dbe7d09637de235667d3cbba35c3d
Apr 02 10:23:43 ethel lighthouse[1511]: Apr 02 17:23:43.663 WARN Batch failed to download. Dropping chain scoring peers, batch_epoch: 42014, score_adjustment: Low Tolerance Error, chain: 11280978360269843115, service: sync
Apr 02 10:23:43 ethel lighthouse[1511]: Apr 02 17:23:43.663 WARN Chain failed! Syncing to its head won't be retried for at least the next 30 seconds, id: 11280978360269843115, from: 42013, to: 42125, end_root: 0x3972…7172, current_target: 42014, batches: 8, peers: 0, validated_batches: 0, state: Syncing, service: sync
Apr 02 10:23:43 ethel lighthouse[1511]: Apr 02 17:23:43.665 INFO Sync state updated                      new_state: Evaluating known peers, old_state: Syncing Finalized Chain, service: sync
Apr 02 10:23:54 ethel lighthouse[1511]: Apr 02 17:23:54.000 INFO Syncing                                 est_time: --, distance: 3599 slots (11 hrs 59 mins), peers: 38, service: slot_notifier
Apr 02 10:23:54 ethel lighthouse[1511]: Apr 02 17:23:54.000 WARN Syncing deposit contract block cache    est_blocks_remaining: 2118, service: slot_notifier
Apr 02 10:24:06 ethel lighthouse[1511]: Apr 02 17:24:06.001 INFO Syncing                                 est_time: --, distance: 3600 slots (12 hrs 0 mins), peers: 37, service: slot_notifier
Apr 02 10:24:06 ethel lighthouse[1511]: Apr 02 17:24:06.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 2118, service: slot_notifier
Apr 02 10:24:06 ethel lighthouse[1511]: Apr 02 17:24:06.303 INFO Address updated                         udp_port: 18270, ip: 23.119.164.150, service: libp2p
Apr 02 10:24:18 ethel lighthouse[1511]: Apr 02 17:24:18.000 INFO Syncing                                 est_time: --, distance: 3601 slots (12 hrs 0 mins), peers: 38, service: slot_notifier
Apr 02 10:24:18 ethel lighthouse[1511]: Apr 02 17:24:18.000 WARN Syncing deposit contract block cache    est_blocks_remaining: 2118, service: slot_notifier
Apr 02 10:24:20 ethel lighthouse[1511]: Apr 02 17:24:20.142 WARN Execution endpoint is not synced        last_seen_block_unix_timestamp: 1712034996, endpoint: http://127.0.0.1:8551/, auth=true, service: deposit_contract_rpc
Apr 02 10:24:20 ethel lighthouse[1511]: Apr 02 17:24:20.142 ERRO Error updating deposit contract cache   error: Failed to get remote head and new block ranges: EndpointError(FarBehind), retry_millis: 60000, service: deposit_contract_rpc
Apr 02 10:24:25 ethel systemd[1]: Stopping eth2 beacon chain service...
Apr 02 10:24:25 ethel lighthouse[1511]: Apr 02 17:24:25.391 INFO Shutting down..                         reason: Success("Received SIGTERM")

And here is the validator:

Apr 02 10:23:59 ethel lighthouse[1513]: Apr 02 17:23:59.003 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3599
Apr 02 10:24:00 ethel lighthouse[1513]: Apr 02 17:24:00.004 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3600
Apr 02 10:24:06 ethel lighthouse[1513]: Apr 02 17:24:06.000 ERRO No synced beacon nodes                  synced: 0, available: 1, total: 1, service: notifier
Apr 02 10:24:06 ethel lighthouse[1513]: Apr 02 17:24:06.000 INFO All validators active                   slot: 1348020, epoch: 42125, total_validators: 2, active_validators: 2, current_epoch_proposers: 0, service: notifier
Apr 02 10:24:10 ethel lighthouse[1513]: Apr 02 17:24:10.001 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3600
Apr 02 10:24:11 ethel lighthouse[1513]: Apr 02 17:24:11.003 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3600
Apr 02 10:24:12 ethel lighthouse[1513]: Apr 02 17:24:12.005 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3601
Apr 02 10:24:18 ethel lighthouse[1513]: Apr 02 17:24:18.000 ERRO No synced beacon nodes                  synced: 0, available: 1, total: 1, service: notifier
Apr 02 10:24:18 ethel lighthouse[1513]: Apr 02 17:24:18.000 INFO All validators active                   slot: 1348021, epoch: 42125, total_validators: 2, active_validators: 2, current_epoch_proposers: 0, service: notifier
Apr 02 10:24:22 ethel lighthouse[1513]: Apr 02 17:24:22.001 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3601
Apr 02 10:24:23 ethel lighthouse[1513]: Apr 02 17:24:23.004 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3601
Apr 02 10:24:24 ethel lighthouse[1513]: Apr 02 17:24:24.007 WARN Beacon node is not synced               el_offline: false, endpoint: http://localhost:5052/, head_slot: 1344420, sync_distance: 3602
Apr 02 10:24:25 ethel systemd[1]: Stopping eth2 validator service...
Apr 02 10:24:25 ethel lighthouse[1513]: Apr 02 17:24:25.393 INFO Shutting down..                         reason: Success("Received SIGTERM")

That didn’t help so after waiting a short while, I decided reboot the computer.

Apr 02 10:44:24 ethel reth[44917]: 2024-04-02T17:44:24.560383Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:44:49 ethel reth[44917]: 2024-04-02T17:44:49.560744Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:45:14 ethel reth[44917]: 2024-04-02T17:45:14.560489Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:45:39 ethel reth[44917]: 2024-04-02T17:45:39.560018Z  INFO Status connected_peers=16 freelist=114325 latest_block=1264352
Apr 02 10:45:39 ethel systemd[1]: Stopping reth eth2 service...
Apr 02 10:45:39 ethel reth[44917]: 2024-04-02T17:45:39.636940Z  INFO Wrote network peers to file peers_file="/home/kendall/eth2-data/reth/holesky/known-peers.json"
Apr 02 10:45:40 ethel systemd[1]: reth.service: Deactivated successfully.
Apr 02 10:45:40 ethel systemd[1]: Stopped reth eth2 service.
Apr 02 10:45:40 ethel systemd[1]: reth.service: Consumed 5h 34min 54.828s CPU time.
-- Boot 88e50cddb9e2424aa5b3d28d32172133 --
Apr 02 10:47:11 ethel systemd[1]: Starting reth eth2 service...
Apr 02 10:47:13 ethel systemd[1]: Started reth eth2 service.
Apr 02 10:47:13 ethel reth[1523]: 2024-04-02T17:47:13.397670Z  INFO Starting reth version="0.2.0-beta.4 (c04dbe6e9)"
Apr 02 10:47:13 ethel reth[1523]: 2024-04-02T17:47:13.399483Z  INFO Opening database path="/home/kendall/eth2-data/reth/holesky/db"
Apr 02 10:47:13 ethel reth[1523]: 2024-04-02T17:47:13.432587Z  INFO Configuration loaded path="/home/kendall/eth2-data/reth/holesky/reth.toml"
Apr 02 10:47:13 ethel reth[1523]: 2024-04-02T17:47:13.441092Z  INFO Database opened
Apr 02 10:47:13 ethel reth[1523]: 2024-04-02T17:47:13.441467Z  INFO Starting metrics endpoint addr=0.0.0.0:9001

And reth found a problem Apr 02 10:49:48 ethel reth[1523]: shutting down due to error and restarted Apr 02 10:49:51 ethel systemd[1]: reth.service: Scheduled restart job, restart counter is at 1:

Apr 02 10:49:46 ethel reth[1523]: 2024-04-02T17:49:46.276911Z  INFO Executing stage pipeline_stages=7/12 stage=StorageHashing checkpoint=1264352 target=1264353
Apr 02 10:49:46 ethel reth[1523]: 2024-04-02T17:49:46.306517Z  INFO Stage finished executing pipeline_stages=7/12 stage=StorageHashing checkpoint=1264353 target=1264353 stage_progress=100.00%
Apr 02 10:49:46 ethel reth[1523]: 2024-04-02T17:49:46.325524Z  INFO Preparing stage pipeline_stages=8/12 stage=MerkleExecute checkpoint=1264352 target=1264353
Apr 02 10:49:46 ethel reth[1523]: 2024-04-02T17:49:46.325567Z  INFO Executing stage pipeline_stages=8/12 stage=MerkleExecute checkpoint=1264352 target=1264353
Apr 02 10:49:46 ethel reth[1523]: 2024-04-02T17:49:46.622428Z  INFO Status connected_peers=3 freelist=114673 stage=MerkleExecute checkpoint=1264352 target=1264353
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.143733Z  WARN Failed to verify block state root target_block=1264353 got=0xa34bd0e7cebbf20708fbae4423721a00106e19e2fd075818f05d0582df314694 expected=SealedHeader { hash: 0x168c92f7f36f0d968ee9f0353268af64c0e987e098f0b29d9d99483f1f6e4af2, header: Header { parent_hash: 0x4fe60b1e2aaccc7e25006671727879e2a2535fa087231efaa0424821bf9fa57e, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x670b24610df99b1685aeac0dfd5307b92e0cf4d7, state_root: 0x4ed32b88c105a497378e0773cf741fcac17ab4474b074f515e0ad7f99dbb4af8, transactions_root: 0xf1fe1dcd01a95693a4f7ad6117c2ab3e02c9f3972cfafe9536ea15f3fe684be6, receipts_root: 0x593c222ab728aecf7647bf67d937fbd426240452a2f54ec75553be018f0fdd6e, withdrawals_root: Some(0xc83f2b05640ff902ff8b1d4bc6d1e75c7d26033c47e70545ff71bdbed577b7cc), logs_bloom: 0x84000080080000002000000000000004400000000400000004000000010040000080010000000200002520800040100000002001004000000014000010200000000000800000000000000089000000004110000880080010000404190880002002000000020000003080000000000900000000000000102804001010006002100004000002000000100000400000000400002000000600000000000000000400030008000000000020800000050000800200000000800010080080240000000000180002000000044000000480100510000100020000200200081000400071400010020000000000000001020008000008000000021085000000000000000000, difficulty: 0x0_U256, number: 1264353, gas_limit: 30000000, gas_used: 1748528, timestamp: 1712035008, mix_hash: 0xf8bbdbe7d817b64fe6719bf76c705c6a5e4e24fbb24014e83a8b5d7dedc76bbd, nonce: 0, base_fee_per_gas: Some(10), blob_gas_used: Some(0), excess_blob_gas: Some(79691776), parent_beacon_block_root: Some(0x7b8202f9b26aba518b6ed53f2683b87d8ba8a2b33f2a6cca6a76a3a53715e094), extra_data: 0x4e65746865726d696e642d312e32352e34 } }
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.143823Z ERROR Stage encountered a validation error: mismatched block state root: got 0xa34bd0e7cebbf20708fbae4423721a00106e19e2fd075818f05d0582df314694, expected 0x4ed32b88c105a497378e0773cf741fcac17ab4474b074f515e0ad7f99dbb4af8 stage=MerkleExecute bad_block=1264353
Apr 02 10:49:48 ethel reth[1523]: Stage encountered a validation error: mismatched block state root: got 0xa34bd0e7cebbf20708fbae4423721a00106e19e2fd075818f05d0582df314694, expected 0x4ed32b88c105a497378e0773cf741fcac17ab4474b074f515e0ad7f99dbb4af8
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.154152Z  INFO Unwinding{stage=StorageHashing}: Stage unwound stage=StorageHashing unwind_to=1264352 progress=1264352 done=true
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.191349Z  INFO Unwinding{stage=AccountHashing}: Stage unwound stage=AccountHashing unwind_to=1264352 progress=1264352 done=true
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.219529Z  WARN Unwinding{stage=MerkleUnwind}: Failed to verify block state root target_block=1264352 got=0xd12b335d07b9f7ae1f87b046a6ee8fc5545752beb0766adf8c813c20c204bd4c expected=SealedHeader { hash: 0x4fe60b1e2aaccc7e25006671727879e2a2535fa087231efaa0424821bf9fa57e, header: Header { parent_hash: 0x905f2d086807cf2b3c03a775fd41e9cb82f86b4782a14d51012797353be63361, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x1268ad189526ac0b386faf06effc46779c340ee6, state_root: 0x2e7b03a91d5318d3f03889cd0190b6211af21766db7aa2b36bbf6daab003b8f4, transactions_root: 0x49bc5b0b00362ca4945e00ee6c6919c61a5bd8af511bca6e25aa1017bf90783e, receipts_root: 0x7fc97e9ad3aa07c4536806fbb831c20c02bcffbe9b588e9ce7c7ae5237483517, withdrawals_root: Some(0x9768a3c497fd628b5f8060a84ba284f26e315b92f677518940e6b225d920e62b), logs_bloom: 0x20000000080000000000000000800004400000000400000004000000004040000000000040000010000420000000000010042001800000000006000000200000000000000000000000040009000048000110000000080004609400090080000010000010000004000000000000000100000000000000002000000010002000001000000010002000000000000000000000002000000000000200000008000400830008000000000000000000010008800000200000040000100000040000000000080002000001004000000400000500080000020000000000001000000001500010000000020000001001008008000008000000000001000000000000000000, difficulty: 0x0_U256, number: 1264352, gas_limit: 30000000, gas_used: 2073135, timestamp: 1712034996, mix_hash: 0xc3271283869684f4f85203cc0688faa9eff2b8fd1800650eebde165115cfdece, nonce: 0, base_fee_per_gas: Some(11), blob_gas_used: Some(786432), excess_blob_gas: Some(79298560), parent_beacon_block_root: Some(0xfa17b7d943717425f130b0e5010a5692c0f34770f8158eaacf6adc636ff380e6), extra_data: 0xd883010d0d846765746888676f312e32312e37856c696e7578 } }
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.224924Z ERROR shutting down due to error
Apr 02 10:49:48 ethel reth[1523]: shutting down due to error
Apr 02 10:49:48 ethel reth[1523]: 2024-04-02T17:49:48.226501Z  INFO Wrote network peers to file peers_file="/home/kendall/eth2-data/reth/holesky/known-peers.json"
Apr 02 10:49:48 ethel reth[1523]: Error: stage encountered an error in block #1264352: validation error: mismatched block state root: got 0xd12b335d07b9f7ae1f87b046a6ee8fc5545752beb0766adf8c813c20c204bd4c, expected 0x2e7b03a91d5318d3f03889cd0190b6211af21766db7aa2b36bbf6daab003b8f4
Apr 02 10:49:48 ethel reth[1523]: Caused by:
Apr 02 10:49:48 ethel reth[1523]:    0: validation error: mismatched block state root: got 0xd12b335d07b9f7ae1f87b046a6ee8fc5545752beb0766adf8c813c20c204bd4c, expected 0x2e7b03a91d5318d3f03889cd0190b6211af21766db7aa2b36bbf6daab003b8f4
Apr 02 10:49:48 ethel reth[1523]:    1: mismatched block state root: got 0xd12b335d07b9f7ae1f87b046a6ee8fc5545752beb0766adf8c813c20c204bd4c, expected 0x2e7b03a91d5318d3f03889cd0190b6211af21766db7aa2b36bbf6daab003b8f4
Apr 02 10:49:48 ethel reth[1523]: Location:
Apr 02 10:49:48 ethel reth[1523]:     /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/task/poll.rs:255:39
Apr 02 10:49:48 ethel systemd[1]: reth.service: Main process exited, code=exited, status=1/FAILURE
Apr 02 10:49:48 ethel systemd[1]: reth.service: Failed with result 'exit-code'.
Apr 02 10:49:48 ethel systemd[1]: reth.service: Consumed 7.614s CPU time.
Apr 02 10:49:51 ethel systemd[1]: reth.service: Scheduled restart job, restart counter is at 1.
Apr 02 10:49:51 ethel systemd[1]: Stopped reth eth2 service.
Apr 02 10:49:51 ethel systemd[1]: reth.service: Consumed 7.614s CPU time.
Apr 02 10:49:51 ethel systemd[1]: Starting reth eth2 service...
Apr 02 10:49:53 ethel systemd[1]: Started reth eth2 service.
Apr 02 10:49:53 ethel reth[2179]: 2024-04-02T17:49:53.430138Z  INFO Starting reth version="0.2.0-beta.4 (c04dbe6e9)"
Apr 02 10:49:53 ethel reth[2179]: 2024-04-02T17:49:53.430853Z  INFO Opening database path="/home/kendall/eth2-data/reth/holesky/db"
Apr 02 10:49:53 ethel reth[2179]: 2024-04-02T17:49:53.448014Z  INFO Configuration loaded path="/home/kendall/eth2-data/reth/holesky/reth.toml"
Apr 02 10:49:53 ethel reth[2179]: 2024-04-02T17:49:53.449092Z  INFO Database opened
Apr 02 10:49:53 ethel reth[2179]: 2024-04-02T17:49:53.449128Z  INFO Starting metrics endpoint addr=0.0.0.0:9001
Apr 02 10:49:53 ethel reth[2179]: 2024-04-02T17:49:53.449307Z  INFO

And kept restarting:

Apr 02 10:49:48 ethel reth[1523]: shutting down due to error
...
Apr 02 10:49:51 ethel systemd[1]: reth.service: Scheduled restart job, restart counter is at 1
...
Apr 02 10:49:53 ethel reth[2179]: shutting down due to error
...
Apr 02 10:49:56 ethel systemd[1]: reth.service: Scheduled restart job, restart counter is at 2.
...
Apr 02 10:49:59 ethel reth[2212]: shutting down due to error
...
Apr 02 10:50:02 ethel systemd[1]: reth.service: Scheduled restart job, restart counter is at 3.
...

Node logs

I'll provide a link to the log files shortly.

Platform(s)

Linux (x86)

What version/commit are you on?

kendall@ethel 24-04-02T19:30:56.169Z:~
$ printf "\nreth --version:\n" && ~/bin/reth --version && printf "\nlighthouse --version:\n" && lighthouse --version && printf "\ninxi:\n" && inxi && printf "\nlsb_release -a:\n" &&  lsb_release -a && printf "\ndf -h:\n" && df -h

reth --version:
reth Version: 0.2.0-beta.4
Commit SHA: c04dbe6e9
Build Timestamp: 2024-03-27T17:50:28.869727319Z
Build Features: jemalloc
Build Profile: release

lighthouse --version:
Lighthouse v5.1.3-3058b96
BLS library: blst-modern
SHA256 hardware acceleration: false
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (true)

inxi:
CPU: quad core Intel Core i5-8259U (-MT MCP-) speed/min/max: 426/400/3800 MHz
Kernel: 6.5.0-26-generic x86_64 Up: 1h 43m Mem: 1322.1/31953.5 MiB (4.1%)
Storage: 2.73 TiB (7.8% used) Procs: 250 Shell: Bash inxi: 3.3.13

lsb_release -a:
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

df -h:
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.2G  1.9M  3.2G   1% /run
/dev/nvme0n1p2  916G   60G  810G   7% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
efivarfs        192K   59K  129K  32% /sys/firmware/efi/efivars
/dev/nvme0n1p1  511M  6.1M  505M   2% /boot/efi
/dev/sda1       1.8T  158G  1.6T  10% /home/kendall/eth2-data
tmpfs           3.2G  108K  3.2G   1% /run/user/125
tmpfs           3.2G   96K  3.2G   1% /run/user/1000
kendall@ethel 24-04-02T19:30:59.134Z:~

What database version are you on?

kendall@ethel 24-04-02T19:30:59.134Z:~
$ reth db version
Current database version: 2
Local database is uninitialized
kendall@ethel 24-04-02T19:32:13.332Z:~

What type of node are you running?

Archive (default)

What prune config do you use, if any?

There is no [prune] section in reth.toml:

kendall@ethel 24-04-02T19:33:46.840Z:~
$ cat eth2-data/reth/holesky/reth.toml 
[stages.headers]
downloader_max_concurrent_requests = 100
downloader_min_concurrent_requests = 5
downloader_max_buffered_responses = 100
downloader_request_limit = 1000
commit_threshold = 10000

[stages.bodies]
downloader_request_limit = 200
downloader_stream_batch_size = 1000
downloader_max_buffered_blocks_size_bytes = 2147483648
downloader_min_concurrent_requests = 5
downloader_max_concurrent_requests = 100

[stages.sender_recovery]
commit_threshold = 5000000

[stages.execution]
max_blocks = 500000
max_changes = 5000000
max_cumulative_gas = 1500000000000
max_duration = "10m"

[stages.account_hashing]
clean_threshold = 500000
commit_threshold = 100000

[stages.storage_hashing]
clean_threshold = 500000
commit_threshold = 100000

[stages.merkle]
clean_threshold = 50000

[stages.transaction_lookup]
chunk_size = 5000000

[stages.index_account_history]
commit_threshold = 100000

[stages.index_storage_history]
commit_threshold = 100000

[peers]
refill_slots_interval = "5s"
trusted_nodes = []
connect_trusted_nodes_only = false
max_backoff_count = 5
ban_duration = "12h"

[peers.connection_info]
max_outbound = 100
max_inbound = 30
max_concurrent_outbound_dials = 15

[peers.reputation_weights]
bad_message = -16384
bad_block = -16384
bad_transactions = -16384
already_seen_transactions = 0
timeout = -4096
bad_protocol = -2147483648
failed_to_connect = -25600
dropped = -4096
bad_announcement = -1024

[peers.backoff_durations]
low = "30s"
medium = "3m"
high = "15m"
max = "1h"

[sessions]
session_command_buffer = 32
session_event_buffer = 260

[sessions.limits]

[sessions.initial_internal_request_timeout]
secs = 20
nanos = 0

[sessions.protocol_breach_request_timeout]
secs = 120
nanos = 0
kendall@ethel 24-04-02T19:34:04.066Z:~

But my reth.service ExecStart is --full:

kendall@ethel 24-04-02T19:35:26.882Z:~
$ cat /etc/systemd/system/reth.service 
# The ethereum service (part of systemd)
# file: /etc/systemd/system/eth1.service

[Unit]
Description     = reth eth2 service
Requires        = network-online.target
After           = network-online.target

[Service]
User            = kendall
ExecStartPre    = /bin/sleep 2
ExecStart       = \
  /home/kendall/bin/reth node \
  --full \
  --chain holesky \
  --instance 1 \
  --max-outbound-peers 16 \
  --max-inbound-peers 16 \
  --http \
  --http.addr 0.0.0.0 \
  --ws \
  --http.api admin,eth,web3,net \
  --datadir /home/kendall/eth2-data/reth/holesky \
  --authrpc.addr 127.0.0.1 \
  --authrpc.port 8551 \
  --authrpc.jwtsecret /home/kendall/eth2-data/secrets/jwtsecret \
  --log.file.format terminal \
  --log.file.filter debug \
  --log.file.directory /home/kendall/eth2-data/reth \
  --log.file.max-size 200 \
  --log.file.max-files 5 \
  --log.journald \
  --log.journald.filter error \
  --color auto \
  --metrics 0.0.0.0:9001 \

Restart         = on-failure
RestartSec      = 3
TimeoutSec      = 300

[Install]
WantedBy    = multi-user.target
kendall@ethel 24-04-02T19:35:38.440Z:~

If you’ve built Reth from source, provide the full command you used

cargo install --locked --path bin/reth --bin reth

Code of Conduct

• I agree to follow the Code of Conduct