lighthouse: Increased frequency of missed attestations post-merge, errors in consensus client
Description
I am experiencing frequent missed attestations since the merge. I have 7 so far, which is definitely abnormal. I suspect that these might be correlated with errors I see in the log below.
Version
Lighthouse v3.1.0 aarch64 binary (NOT portable)
Present Behaviour
I examined the logs of the beacon client around the point of my last missed attestation at slot 4703169, here’s what I see:
3:57.580 ERRO Execution engine call failed error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(8551), path: "/", query: None, fragment: None }, source: TimedOut }), service: exec
Sep 15 17:13:57 validator lighthouse[927]: Sep 15 17:13:57.582 ERRO Error fetching block for peer error: ExecutionLayerErrorPayloadReconstruction(0x7452239120415b44302afd7c8c5b3bc31b09838c253d8ca5f2d6cf4400e30399, EngineError(Api { error: Reqwest(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, >
...
Sep 15 17:14:08 validator lighthouse[927]: Sep 15 17:14:08.003 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x22a340948469b7448d8a7d5c96b23ea0e05d51f2bb95f1b2a9f30da55ee1198f)), service: state_advance
Sep 15 17:14:15 validator lighthouse[927]: Sep 15 17:14:15.227 INFO New block received root: 0xe09d15f3c06aede951698f5166b41835c677f519cfba93a314de45e9c8fefa95, slot: 4703169
Sep 15 17:14:17 validator lighthouse[927]: Sep 15 17:14:17.002 INFO Synced slot: 4703169, block: … empty, epoch: 146974, finalized_epoch: 146972, finalized_root: 0x4ab8…ac6c, exec_hash: 0x4cfb…3a96 (verified), peers: 88, service: slot_notifier
Sep 15 17:14:20 validator lighthouse[927]: Sep 15 17:14:20.148 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x1fc17f075f6ca7b260cd17d278776c05390c7966daa6ce386d994fe9fcdf14d0)), service: state_advance
...
Sep 15 17:14:57 validator lighthouse[927]: Sep 15 17:14:57.006 ERRO Failed to advance head state error: SnapshotCacheLockTimeout, service: state_advance>
logs of the validator for that missed attestation seem normal:
Sep 15 17:14:15 validator lighthouse[928]: Sep 15 17:14:15.355 INFO Successfully published attestations type: unaggregated, slot: 4703169, committee_index: 46, head_block: 0x1fc17f075f6ca7b260cd17d278776c05390c7966daa6ce386d994fe9fcdf14d0, validator_indices: [109040], count: 1, service: attestation
Expected Behaviour
Not produce errors and not miss attestations.
Steps to resolve
N/A
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (5 by maintainers)
@gamell Please try upgrading to Lighthouse v3.1.2 and setting
--prune-payloads false. This will alleviate some of the load on your execution node.