lighthouse: Unable to start the lighthouse after the update

I have been running the tandem of geth and lighthouse since the last hardfork at it worked perfectly. Today the syncing stopped because of the next hardfork so I did the update of the software to the newest versions, both geth and lightouse. Since then I am unable to run the lighthouse. The error I get is:

Apr 13 10:51:06 ns3109353 systemd[1]: Started ETH Beacon.
Apr 13 10:51:06 ns3109353 lighthouse[16707]: Apr 13 10:51:06.555 INFO Logging to file                         path: "/home/eth/.ethereum/beacon/logs/beacon.log"
Apr 13 10:51:06 ns3109353 lighthouse[16707]: Apr 13 10:51:06.556 INFO Lighthouse started                      version: Lighthouse/v4.0.2-rc.0-35d8c98
Apr 13 10:51:06 ns3109353 lighthouse[16707]: Apr 13 10:51:06.556 INFO Configured for network                  name: mainnet
Apr 13 10:51:06 ns3109353 lighthouse[16707]: Apr 13 10:51:06.556 INFO Data directory initialised              datadir: /home/eth/.ethereum
Apr 13 10:51:06 ns3109353 lighthouse[16707]: Apr 13 10:51:06.556 INFO Deposit contract                        address: 0x00000000219ab540356cbb839cbe05303d7705fa, deploy_block: 11184524
Apr 13 10:51:07 ns3109353 lighthouse[16707]: Apr 13 10:51:07.106 INFO Hot-Cold DB initialized                 split_state: 0x173c07e36c2c337a88de6f465f30e4ccef5c3eeca2d220fd13b45e61f5409ccb, split_slot: 6209472, service: freezer_db
Apr 13 10:51:07 ns3109353 lighthouse[16707]: Apr 13 10:51:07.114 INFO Starting beacon chain                   method: resume, service: beacon
Apr 13 10:51:46 ns3109353 lighthouse[16707]: Apr 13 10:51:46.829 INFO Block production enabled                method: json rpc via http, endpoint: Auth { endpoint: "http://127.0.0.1:8551/", jwt_path: "/home/eth/node_beacon/jwt.hex", jwt_id: None, jwt_version: None }
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.733 WARN Execution endpoint is not synced        last_seen_block_unix_timestamp: 1681336943, endpoint: http://127.0.0.1:8551/, auth=true, service: deposit_contract_rpc
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.734 ERRO Error updating deposit contract cache   error: Failed to get remote head and new block ranges: EndpointError(FarBehind), retry_millis: 60000, service: deposit_contract_rpc
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.738 ERRO Error decoding head block               message: This node has likely missed a hard fork. It will try to revert the invalid blocks and keep running, but any stray blocks and states will not be deleted. Long-term you should consider re-syncing this node., service: beacon
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.739 WARN Reverting invalid head block            fork_epoch: 194048, target_fork: capella, service: beacon
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.742 INFO Reverting block                         slot: 6211993, block_root: 0x2081a9b74435a76f9f3d49826491e8071db2a045f3250d84c1f2d88fb6d04057, service: beacon
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.747 INFO Reverting block                         slot: 6211813, block_root: 0xb14a3f641973f6f6a6ff6bb421190cf95309ac556f3fc3eea0957b52b5631d8a, service: beacon
Apr 13 10:51:47 ns3109353 lighthouse[16707]: Apr 13 10:51:47.752 INFO Reverting block                         slot: 6209582, block_root: 0x8bc7203e123cef451aadcaa541970a5c6c61780c3a9550893daecda7a9bc8166, service: beacon
Apr 13 10:51:48 ns3109353 lighthouse[16707]: Apr 13 10:51:48.319 CRIT Failed to start beacon node             reason: Failed to build beacon chain: Error loading finalized block: HotColdDBError(MissingFullBlockExecutionPayloadPruned(0x1171bd9c5f5ed58daf7ac4e07f6a1b7d5e7fb7adc7dde42b9cf9e1735dc884bc, Slot(6209439)))
Apr 13 10:51:48 ns3109353 lighthouse[16707]: Apr 13 10:51:48.319 INFO Internal shutdown received              reason: Failed to start beacon node
Apr 13 10:51:48 ns3109353 lighthouse[16707]: Apr 13 10:51:48.319 INFO Shutting down..                         reason: Failure("Failed to start beacon node")
Apr 13 10:51:48 ns3109353 lighthouse[16707]: Failed to start beacon node

What should I do?

About this issue

Original URL
State: closed
Created a year ago
Comments: 19 (9 by maintainers)

Most upvoted comments

Just make sure you remove the purge-db (after you are up and running again) in your systemd file (and daemon-reload it)… or else the next time you restart the process it will blast the db and checkpoint sync again… not the end of the world, but not needed.

mrabino1 on Apr 15, 2023

You need to add purge-db, because your database already exists

omg omg omg You are so nice It’s really useful!!! LOG: Syncing, est_time: --, distance: 117 slots (23 mins), peers: 1, service: slot_notifier, module: client::notifier:250 Thank you very very much!!! You saved me a lot of days!!

llh3719 on Apr 15, 2023

@jolestar Why don’t you just update to the latest version before the fork?

Just forgot😅, I am re-syncing.

jolestar on Apr 17, 2023

We’ll try to fix this for the next fork, but for the current one I think you need to re-sync. Sorry.

michaelsproul on Apr 17, 2023

I did the suggested --purge-db with --checkpoint-sync-url and it was able to catch up with the mainnet. After doing that I restarted the service without those two flags and I is able to sync on its own now properly. That solved the issue. Thank you.

khelle on Apr 13, 2023

likely need to purge the db. (–purge-db) … and then checkpoint sync to get back going…

mrabino1 on Apr 13, 2023