lighthouse: v5.2.0 built from source runs out of memory
Description
We build a Docker image from the Lighthouse v5.2.0 Dockerfile with very minor changes:
- adding
curlinside the image directly - some changes relating to how the binary is run
The binary itself is built the same way as in this repo’s Dockerfile, using make and the default values for FEATURES, PROFILE, etc…
diff --git a/Dockerfile b/Dockerfile
index e0c48699b..da033a4bf 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -13,6 +13,21 @@ FROM ubuntu:22.04
RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-recommends \
libssl-dev \
ca-certificates \
- && apt-get clean \
- && rm -rf /var/lib/apt/lists/*
+ curl && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
COPY --from=builder /usr/local/cargo/bin/lighthouse /usr/local/bin/lighthouse
+RUN useradd -m -u 1000 -s /bin/bash lighthouse
+RUN chown -R lighthouse:lighthouse /opt
+USER lighthouse
+RUN mkdir /opt/data
+WORKDIR /opt/lighthouse
+
+# Firewall configurations
+# P2P TCP, UDP
+EXPOSE 9001/TCP
+EXPOSE 9001/UDP
+# RPC
+EXPOSE 5053
+ENTRYPOINT [ "lighthouse" ]
The node is run using these arguments:
--datadir=/opt/data
--http
--listen-address=0.0.0.0
--listen-address=::
--http-address=::
--http-allow-origin=*
--http-port=5052
--target-peers=50
--slots-per-restore-point=32
--prune-blobs=false
--execution-jwt=/secrets/jwtsecret
--execution-endpoint=http://....:8551
--port=9000
--port6=9000
--network=mainnet
--disable-deposit-contract-sync
--state-cache-size=2
--metrics
--metrics-address=::
--metrics-port=8080
--metrics-allow-origin=*
Version
The Docker image is built from v5.2.0
Present Behaviour
Once started, the application runs then finally runs out of memory after hitting the limit of 64GB we assigned to it.
This did not happen with the same modifications applied to previous versions of Lighthouse. The last one we had tested this with was v5.0.0.
Using the image you provide (sigp/lighthouse:v5.2.0) with the same arguments on the same datadir results in a reasonable, stable, memory usage.
Expected Behaviour
The application should have a stable memory footprint and not run out of memory.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (12 by maintainers)
We have another report by @rz4884 on Discord that’s facing the same issue, and the user confirmed that it is because he doesn’t include
--feature jemalloc.@michaelsproul mentions that the Dockerfile appears to not have enabled
jemallocby default. We will fix this, thanks for reportingI wonder if
maxperfis causing the compiler to find some optimisation that prevents an OOM. I will try to repro and profile the mem usage of areleasebuild.Please send me debug logs (
@sproulon Discord) and I’ll also open a new issue once we have some idea of what the problem is