lighthouse: BlockDownloadFailed (TimedOut) error downloading Eth1 blocks [Infura, Alchemy, local Geth]

Description

My validator has been running on mainnet for a few days now. I’m syncing a local Geth client in the background for the long haul, but for now my Beacon node has been hooked up to Infura. I successfully made deposits and my validators are active and attesting, so the ETH1 endpoint was working at some point. But in the past day I’ve been seeing the following…

May 27 16:22:00 ethereum lighthouse[165241]: May 27 16:22:00.198 ERRO Failed to update eth1 cache error: Failed to update Eth1 service: "All fallback errored: https://mainnet.infura.io/v3/xxx => BlockDownloadFailed(\"Request failed: reqwest::Error { kind: Request, url: Url { scheme: \\\"https\\\", username: \\\"\\\", password: None, host: Some(Domain(\\\"mainnet.infura.io\\\")), port: None, path: \\\"/v3/xxx\\\", query: None, fragment: None }, source: TimedOut }\")", retry_millis: 7000, service: eth1_rpc

I thought that maybe Infura was rate-limiting me, but when I log into Infura I see in their dashboard that I’m both well below the limit and also that the node appears to be successfully making requests.

I signed up for Alchemy to try another provider and I’m seeing the same behavior. Requests are made, Alchemy’s dashboard shows the requests are happening and responding in the 10ms range.

Then I hooked up Lighthouse to the local Geth (even though it’s not done syncing), and saw the same TimedOut behavior even on localhost, which seems unlikely.

I can curl Infura and Alchemy and Geth just fine. Infura responds to POST with JSON. Curl doesn’t time out. This leads me to believe that there is some sort of issue with the Rust client itself.

Version

Lighthouse v1.3.0-3a24ca5
BLS Library: blst-modern
Specs: mainnet (true), minimal (false), v0.12.3 (false)

Local Geth is

1.10.3-stable-991384a7

Local Geth sync progress is currently:

{
  currentBlock: 12518158,
  highestBlock: 12518277,
  knownStates: 732082090,
  pulledStates: 731954144,
  startingBlock: 12514158
}

I have no idea what the remote nodes are for Infura or Alchemy.

Present Behaviour

Eth1 syncing is timing out with multiple providers, even when response times seem reasonable. Connections appear to be initiating but not resolving in the rust client.

Expected Behaviour

Eth1 endpoints should be queried. Replies should be parsed and cached.

Steps to resolve

Unknown.