static-web-server: SWS fails to start on Raspberry Pi 5 with the message "Unsupported system page size"
Search for duplicate issues
- I already searched, and this issue is not a duplicate.
Issue scope
Docker / Kubernetes
Describe the bug
Hi,
When I try to deploy SWS on a K3S kubernetes server running on a Raspberry Pi 5, the containers fail to start with the following error message:
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Kubernetes information on the container (information anonymized with XXX):
kubectl describe pod sws-deployment-66f5b8b9b8-z9vj8
Name: sws-deployment-66f5b8b9b8-z9vj8
Namespace: XXXXX
Priority: 0
Service Account: default
Node: XXXXX/192.168.XXX.XXX
Start Time: Sat, 11 May 2024 22:08:49 +0200
Labels: app=sws
pod-template-hash=66f5b8b9b8
Annotations: <none>
Status: Running
IP: 10.42.1.26
IPs:
IP: 10.42.1.26
Controlled By: ReplicaSet/sws-deployment-66f5b8b9b8
Containers:
sws:
Container ID: containerd://3f227308244481d87f40679cf9c1722654a27ba61e4a65ef90065e3ef1975e6f
Image: joseluisq/static-web-server:2.30.0-debian
Image ID: docker.io/joseluisq/static-web-server@sha256:c88844fd56f7462c2e0818d49df4dfd3e0ebdb12139e81d72c186df56c97d6a7
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 139
Started: Sat, 11 May 2024 22:25:03 +0200
Finished: Sat, 11 May 2024 22:25:03 +0200
Ready: False
Restart Count: 8
Environment:
FAKE_DISABLE_JEMALLOC: true
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmbls (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-tmbls:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned XXXXX/sws-deployment-66f5b8b9b8-z9vj8 to XXXX
Normal Pulling 17m kubelet Pulling image "joseluisq/static-web-server:2.30.0-debian"
Normal Pulled 17m kubelet Successfully pulled image "joseluisq/static-web-server:2.30.0-debian" in 6.258s (6.258s including waiting)
Normal Pulled 15m (x4 over 17m) kubelet Container image "joseluisq/static-web-server:2.30.0-debian" already present on machine
Normal Created 15m (x5 over 17m) kubelet Created container sws
Normal Started 15m (x5 over 17m) kubelet Started container sws
Warning BackOff 2m34s (x69 over 17m) kubelet Back-off restarting failed container sws in pod sws-deployment-66f5b8b9b8-z9vj8_XXXXX(d6040825-8420-4f32-9fc2-52389ab1c7db)
The same error is present with the scratch version or the docker image.
I found a bug that has been fixed and could be related to the same root cause on another software: https://github.com/qdrant/qdrant/issues/2474 or another related PR and also https://github.com/typesense/typesense/issues/1351 . I tried to add the DISABLE_JEMALLOC environment variable but it did not fix the issue (Note: then I renamed it to avoid potential conflicts).
For information, the page size seems to be configured to 16K on the system (total of 8Gi of RAM installed).
getconf PAGE_SIZE
16384
Have a nice day.
How to reproduce it
-
Install a k3s server on a Rasberry Pi 5B with 8G of RAM.
-
Deploy SWS using the following deployment file name sws.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sws-deployment
labels:
app: sws
spec:
replicas: 2
selector:
matchLabels:
app: sws
template:
metadata:
labels:
app: sws
spec:
containers:
- name: sws
image: joseluisq/static-web-server:2.30.0-debian
ports:
- containerPort: 80
Command: kubectl apply -f sws.yaml
- Check the pods status with
kubectl get pods
NAME READY STATUS RESTARTS AGE
sws-deployment-66f5b8b9b8-z9vj8 0/1 CrashLoopBackOff 10 (3m41s ago) 30m
sws-deployment-66f5b8b9b8-644jw 0/1 CrashLoopBackOff 10 (3m34s ago) 30m
The pods should be in the Running state but are in the CrashLoopBackOff after a few minutes.
The logs are indicating an issue with jemalloc : kubectl logs sws-deployment-66f5b8b9b8-z9vj8
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Expected behavior
The pods should be in the Running state after the deployment and not error should be logged.
Complementary information
The image deployed seem to be the correct one as the SHA matches the k3s description above: https://hub.docker.com/layers/joseluisq/static-web-server/2.30.0-debian/images/sha256-b7dce340af9f5d282de917dd9ae27306e0620f54377da4b0ce51cf32f1538329?context=explore
The issue seems to be documented on the Raspberry Pi issue tracker: https://github.com/raspberrypi/bookworm-feedback/issues/107
Build target
Docker ~linux/amd64~ linux/arm64
Environment and specs
- static-web-server: 2.30.0-debian
- OS: Linux XXXXX 6.6.28+rpt-rpi-2712 # 1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux
- Arch: ARM64 (64-bit) on Rasberry Pi 5B
- Docker: k3s kubernetes version v1.29.4+k3s1
- Client: Not applicable
- Specify others
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 23
This issue is confirmed to be fixed in release v2.31.1 (see https://github.com/static-web-server/static-web-server/pull/446#issuecomment-2121738796 for details).
@joseluisq, just to confirm @sharknoon feedback, I started the binaries directly on the Pi 5 (outside of Docker/K8S) on both kernels and it starts successfully on them. Thank you very much for your exceptional support!
Ok, exploring a bit more in detail. Turns out that it is a well-known problem with a solution. Basically, we can keep musl static/linked binaries for
aarch64(a.k.a. ARM64) targets but we have to tell jemalloc to use a larger page size by default.So we can go with using the maximal supported page size of
64KBonaarch64which should be not a problem for system with smaller page sizes since jemalloc supports that functionality. See https://github.com/jemalloc/jemalloc/pull/769I will prepare a fix soon.
Thank you @fred00 for testing and @joseluisq for a very fast response
I tried to test the provided binaries outside of kubernetes / container.d to see if they would start. I used
ctrto mount the image directly on the host OS. I get the same error.Thank you! I am really amazed how quickly you responded to this error. The static web server is an amazing software and I really enjoy using it. Thank you for your time and effort, I really appreciate it 😃
Thanks for checking. That’s what I was assuming. I just wanted to double-check.
Now, two things to clarify.
First, regarding the musl static-linked binary for
linux/arm64, we still want to compile jemalloc with a 64KB page size default.Second, regarding the Debian Docker image for
linux/arm64, I’m still thinking on why not provide dynamic-linked binaries (glibc) likeaarch64-unknown-linux-gnufor the Debian-based images only and leave the static-linked binaries (musl) for the Alpine and the Scratch images for consistency. This has to come up later at some point. But for now, we will fix the issue by increasing the page size as discussed in https://github.com/static-web-server/static-web-server/issues/440#issuecomment-2106958346.@joseluisq It this how you imagined the Debian test image? Dockerfile
@joseluisq I’m not sure to have the toolchain required to build a new image for aarch64. If @sharknoon can’t do it I will try my best. If you prepare an image with a fix, I will be able to test it with both Pi 5 kernels to provide you with a feedback on the fix compatibility.
Workaround
After some web search, i found a workaround that could confirm the issue with the page 16K page size ( https://github.com/raspberrypi/bookworm-feedback/issues/107 ). I added the
kernel=kernel8.imgline after the[all]section in/boot/firmware/config.txtfile and rebooted the Raspberry Pi 5.Now
uname -ashow that the kernel type/version changed from 2712 to 8 :The pods are in the Running state:
And the logs show a successful start of SWS: