lighthouse: v5.2.0 built from source runs out of memory

Description

We build a Docker image from the Lighthouse v5.2.0 Dockerfile with very minor changes:

  • adding curl inside the image directly
  • some changes relating to how the binary is run

The binary itself is built the same way as in this repo’s Dockerfile, using make and the default values for FEATURES, PROFILE, etc…

diff --git a/Dockerfile b/Dockerfile
index e0c48699b..da033a4bf 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -13,6 +13,21 @@ FROM ubuntu:22.04
 RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
   libssl-dev \
   ca-certificates \
-  && apt-get clean \
-  && rm -rf /var/lib/apt/lists/*
+  curl && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
+
 COPY --from=builder /usr/local/cargo/bin/lighthouse /usr/local/bin/lighthouse
+RUN useradd -m -u 1000 -s /bin/bash lighthouse
+RUN chown -R lighthouse:lighthouse /opt
+USER lighthouse
+RUN mkdir /opt/data
+WORKDIR /opt/lighthouse
+
+# Firewall configurations
+# P2P TCP, UDP
+EXPOSE 9001/TCP
+EXPOSE 9001/UDP
+# RPC
+EXPOSE 5053
+ENTRYPOINT [ "lighthouse" ]

The node is run using these arguments:

      --datadir=/opt/data
      --http
      --listen-address=0.0.0.0
      --listen-address=::
      --http-address=::
      --http-allow-origin=*
      --http-port=5052
      --target-peers=50
      --slots-per-restore-point=32
      --prune-blobs=false
      --execution-jwt=/secrets/jwtsecret
      --execution-endpoint=http://....:8551
      --port=9000
      --port6=9000
      --network=mainnet
      --disable-deposit-contract-sync
      --state-cache-size=2
      --metrics
      --metrics-address=::
      --metrics-port=8080
      --metrics-allow-origin=*

Version

The Docker image is built from v5.2.0

Present Behaviour

Once started, the application runs then finally runs out of memory after hitting the limit of 64GB we assigned to it.

This did not happen with the same modifications applied to previous versions of Lighthouse. The last one we had tested this with was v5.0.0.

Using the image you provide (sigp/lighthouse:v5.2.0) with the same arguments on the same datadir results in a reasonable, stable, memory usage.

Expected Behaviour

The application should have a stable memory footprint and not run out of memory.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Compiling with --features "portable,gnosis,slasher-lmdb,slasher-mdbx,jemalloc" --profile "maxperf" makes the application not run out of memory

We have another report by @rz4884 on Discord that’s facing the same issue, and the user confirmed that it is because he doesn’t include --feature jemalloc.

@michaelsproul mentions that the Dockerfile appears to not have enabled jemalloc by default. We will fix this, thanks for reporting

I wonder if maxperf is causing the compiler to find some optimisation that prevents an OOM. I will try to repro and profile the mem usage of a release build.

Please send me debug logs (@sproul on Discord) and I’ll also open a new issue once we have some idea of what the problem is