lighthouse: OOM when always-prepare-payload is enabled

I run in an OOM error very frequent and reproducible (every 1-3 days), but only when --always-prepare-payload is used. If not it is running fine for a longer period of time.

Using latest stable version: Lighthouse/v7.0.1-e42406d on Linux/x64

Command used:

/opt/eth/.cargo/bin/lighthouse \
--network mainnet beacon_node \
--checkpoint-sync-url https://sync.invis.tools \
--execution-endpoint http://127.0.0.1:18452 \
--execution-jwt /opt/eth/lighthouse_jwt_secret \
--datadir /opt/eth/lighthouse_data \
--port 19103 \
--disable-deposit-contract-sync \
--disable-upnp \
--disable-backfill-rate-limiting \
--http --http-port 41220 --http-address 0.0.0.0 --target-peers 25 \
--suggested-fee-recipient 0x... \
--always-prepare-payload \
--prepare-payload-lookahead 6000

dmesg:

[4073697.882056] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-4745.scope,task=lighthouse,pid=739684,uid=1001
[4073697.882678] Out of memory: Killed process 739684 (lighthouse) total-vm:108570708kB, anon-rss:55394272kB, file-rss:4348kB, shmem-rss:0kB, UID:1001 pgtables:164768kB oom_score_adj:0
[4073701.617960] oom_reaper: reaped process 739684 (lighthouse), now anon-rss:780kB, file-rss:2024kB, shmem-rss:0kB

log output:

Jun 22 23:22:48.243 INFO Address updated                         udp_port: 41465, ip: xx.xx.241.236, service: libp2p
Jun 22 23:22:48.244 INFO Address updated                         udp_port: 19103, ip: xx.xx.171.214, service: libp2p
Jun 22 23:22:48.245 INFO Address updated                         udp_port: 41465, ip: xx.xx.241.236, service: libp2p
Jun 22 23:22:48.517 INFO New block received                      root: 0x904753c0b02597092ce9eaaa5d7eb67e50583cc570f42a946b1a14c6411db6bb, slot: 11984212
Jun 22 23:22:49.035 INFO Address updated                         udp_port: 19103, ip: xx.xx.171.214, service: libp2p
./lighthouse.sh: line 15: 739684 Killed                  /opt/eth/.cargo/bin/lighthouse --network mainnet beacon_node --checkpoint-sync-url https://sync.invis.tools --execution-endpoint http://127.0.0.1:18452 --execution-jwt /opt/eth/lighthouse_jwt_secret --datadir /mnt/xpg8tb/reth-rbuilder/lighthouse_data --port 19103 --disable-deposit-contract-sync --disable-upnp --disable-backfill-rate-limiting --http --http-port 41220 --http-address 0.0.0.0 --target-peers 25 --suggested-fee-recipient 0x... --always-prepare-payload --prepare-payload-lookahead 6000

It seems there is a spike before it dies, but I can not tell 100% if this is lighthouse: Image

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

With unstable it is all stable now 😃 Many thanks to you all for your support. Going to close this issue as resolved.

No, just using ext4. I will switch to unstable, if the issue persist I can check with heaptrack.

My initial assumption was wrong that it is only happening --always-prepare-payload. Because I missed that the first instance just auto restarts. I will switch to unstable, and report back.