static-web-server: SWS fails to start on Raspberry Pi 5 with the message "Unsupported system page size"

Search for duplicate issues

  • I already searched, and this issue is not a duplicate.

Issue scope

Docker / Kubernetes

Describe the bug

Hi,

When I try to deploy SWS on a K3S kubernetes server running on a Raspberry Pi 5, the containers fail to start with the following error message:

<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed

Kubernetes information on the container (information anonymized with XXX):

kubectl describe pod sws-deployment-66f5b8b9b8-z9vj8
Name:             sws-deployment-66f5b8b9b8-z9vj8
Namespace:        XXXXX
Priority:         0
Service Account:  default
Node:             XXXXX/192.168.XXX.XXX
Start Time:       Sat, 11 May 2024 22:08:49 +0200
Labels:           app=sws
                  pod-template-hash=66f5b8b9b8
Annotations:      <none>
Status:           Running
IP:               10.42.1.26
IPs:
  IP:           10.42.1.26
Controlled By:  ReplicaSet/sws-deployment-66f5b8b9b8
Containers:
  sws:
    Container ID:   containerd://3f227308244481d87f40679cf9c1722654a27ba61e4a65ef90065e3ef1975e6f
    Image:          joseluisq/static-web-server:2.30.0-debian
    Image ID:       docker.io/joseluisq/static-web-server@sha256:c88844fd56f7462c2e0818d49df4dfd3e0ebdb12139e81d72c186df56c97d6a7
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Sat, 11 May 2024 22:25:03 +0200
      Finished:     Sat, 11 May 2024 22:25:03 +0200
    Ready:          False
    Restart Count:  8
    Environment:
      FAKE_DISABLE_JEMALLOC:  true
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmbls (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-tmbls:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned XXXXX/sws-deployment-66f5b8b9b8-z9vj8 to XXXX
  Normal   Pulling    17m                   kubelet            Pulling image "joseluisq/static-web-server:2.30.0-debian"
  Normal   Pulled     17m                   kubelet            Successfully pulled image "joseluisq/static-web-server:2.30.0-debian" in 6.258s (6.258s including waiting)
  Normal   Pulled     15m (x4 over 17m)     kubelet            Container image "joseluisq/static-web-server:2.30.0-debian" already present on machine
  Normal   Created    15m (x5 over 17m)     kubelet            Created container sws
  Normal   Started    15m (x5 over 17m)     kubelet            Started container sws
  Warning  BackOff    2m34s (x69 over 17m)  kubelet            Back-off restarting failed container sws in pod sws-deployment-66f5b8b9b8-z9vj8_XXXXX(d6040825-8420-4f32-9fc2-52389ab1c7db)

The same error is present with the scratch version or the docker image.

I found a bug that has been fixed and could be related to the same root cause on another software: https://github.com/qdrant/qdrant/issues/2474 or another related PR and also https://github.com/typesense/typesense/issues/1351 . I tried to add the DISABLE_JEMALLOC environment variable but it did not fix the issue (Note: then I renamed it to avoid potential conflicts).

For information, the page size seems to be configured to 16K on the system (total of 8Gi of RAM installed).

getconf PAGE_SIZE                           
16384

Have a nice day.

How to reproduce it

  1. Install a k3s server on a Rasberry Pi 5B with 8G of RAM.

  2. Deploy SWS using the following deployment file name sws.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sws-deployment
  labels:
    app: sws
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sws
  template:
    metadata:
      labels:
        app: sws
    spec:
      containers:
      - name: sws
        image: joseluisq/static-web-server:2.30.0-debian
        ports:
        - containerPort: 80

Command: kubectl apply -f sws.yaml

  1. Check the pods status with kubectl get pods
NAME                              READY   STATUS             RESTARTS         AGE
sws-deployment-66f5b8b9b8-z9vj8   0/1     CrashLoopBackOff   10 (3m41s ago)   30m
sws-deployment-66f5b8b9b8-644jw   0/1     CrashLoopBackOff   10 (3m34s ago)   30m

The pods should be in the Running state but are in the CrashLoopBackOff after a few minutes.

The logs are indicating an issue with jemalloc : kubectl logs sws-deployment-66f5b8b9b8-z9vj8

<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed

Expected behavior

The pods should be in the Running state after the deployment and not error should be logged.

Complementary information

The image deployed seem to be the correct one as the SHA matches the k3s description above: https://hub.docker.com/layers/joseluisq/static-web-server/2.30.0-debian/images/sha256-b7dce340af9f5d282de917dd9ae27306e0620f54377da4b0ce51cf32f1538329?context=explore

The issue seems to be documented on the Raspberry Pi issue tracker: https://github.com/raspberrypi/bookworm-feedback/issues/107

Build target

Docker ~linux/amd64~ linux/arm64

Environment and specs

  • static-web-server: 2.30.0-debian
  • OS: Linux XXXXX 6.6.28+rpt-rpi-2712 # 1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux
  • Arch: ARM64 (64-bit) on Rasberry Pi 5B
  • Docker: k3s kubernetes version v1.29.4+k3s1
  • Client: Not applicable
  • Specify others

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 23

Most upvoted comments

This issue is confirmed to be fixed in release v2.31.1 (see https://github.com/static-web-server/static-web-server/pull/446#issuecomment-2121738796 for details).

@joseluisq, just to confirm @sharknoon feedback, I started the binaries directly on the Pi 5 (outside of Docker/K8S) on both kernels and it starts successfully on them. Thank you very much for your exceptional support!

$ uname -a
Linux XXX 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

$ ./static-web-server -p 1234 
2024-05-13T16:55:50.293000Z  INFO static_web_server::info: static-web-server 2.30.0
2024-05-13T16:55:50.293030Z  INFO static_web_server::info: log level: error
2024-05-13T16:55:50.293071Z  INFO static_web_server::info: server bound to tcp socket [::]:1234
2024-05-13T16:55:50.293078Z  INFO static_web_server::info: runtime worker threads: 4
2024-05-13T16:55:50.293082Z  INFO static_web_server::info: runtime max blocking threads: 512
2024-05-13T16:55:50.293097Z  INFO static_web_server::info: log remote address: enabled=false
2024-05-13T16:55:50.293102Z  INFO static_web_server::info: redirect trailing slash: enabled=true
2024-05-13T16:55:50.293107Z  INFO static_web_server::info: ignore hidden files: enabled=false
2024-05-13T16:55:50.293111Z  INFO static_web_server::info: grace period before graceful shutdown: 0s
2024-05-13T16:55:50.293119Z  INFO static_web_server::info: index files: index.html
2024-05-13T16:55:50.293124Z  INFO static_web_server::info: directory listing: enabled=false
2024-05-13T16:55:50.293129Z  INFO static_web_server::info: directory listing order code: 6
2024-05-13T16:55:50.293133Z  INFO static_web_server::info: directory listing format: Html
2024-05-13T16:55:50.293141Z  INFO static_web_server::info: fallback page: enabled=false, value=""
2024-05-13T16:55:50.293146Z  INFO static_web_server::info: health endpoint: enabled=false
2024-05-13T16:55:50.293150Z  INFO static_web_server::info: metrics endpoint (experimental): enabled=false
2024-05-13T16:55:50.293156Z  INFO static_web_server::info: basic authentication: enabled=false
2024-05-13T16:55:50.293160Z  INFO static_web_server::info: maintenance mode: enabled=false
2024-05-13T16:55:50.293164Z  INFO static_web_server::info: maintenance mode status: 503
2024-05-13T16:55:50.293169Z  INFO static_web_server::info: maintenance mode file: ""
2024-05-13T16:55:50.293173Z  INFO static_web_server::info: compression static: enabled=false
2024-05-13T16:55:50.293177Z  INFO static_web_server::info: auto compression: enabled=true, formats=deflate,gzip,brotli,zstd
2024-05-13T16:55:50.293182Z  INFO static_web_server::info: cache control headers: enabled=true
2024-05-13T16:55:50.293186Z  INFO static_web_server::info: security headers: enabled=false
2024-05-13T16:55:50.293242Z  INFO static_web_server::info: http1 server is listening on http://[::]:1234
2024-05-13T16:55:50.293259Z  INFO static_web_server::info: press ctrl+c to shut down the server
uname -a
Linux XXX 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

$ ./static-web-server -p 1234 
2024-05-13T16:51:05.036837Z  INFO static_web_server::info: static-web-server 2.30.0
2024-05-13T16:51:05.036865Z  INFO static_web_server::info: log level: error
2024-05-13T16:51:05.036904Z  INFO static_web_server::info: server bound to tcp socket [::]:1234
2024-05-13T16:51:05.036910Z  INFO static_web_server::info: runtime worker threads: 4
2024-05-13T16:51:05.036914Z  INFO static_web_server::info: runtime max blocking threads: 512
2024-05-13T16:51:05.036930Z  INFO static_web_server::info: log remote address: enabled=false
2024-05-13T16:51:05.036935Z  INFO static_web_server::info: redirect trailing slash: enabled=true
2024-05-13T16:51:05.036940Z  INFO static_web_server::info: ignore hidden files: enabled=false
2024-05-13T16:51:05.036944Z  INFO static_web_server::info: grace period before graceful shutdown: 0s
2024-05-13T16:51:05.036954Z  INFO static_web_server::info: index files: index.html
2024-05-13T16:51:05.036960Z  INFO static_web_server::info: directory listing: enabled=false
2024-05-13T16:51:05.036964Z  INFO static_web_server::info: directory listing order code: 6
2024-05-13T16:51:05.036969Z  INFO static_web_server::info: directory listing format: Html
2024-05-13T16:51:05.036979Z  INFO static_web_server::info: fallback page: enabled=false, value=""
2024-05-13T16:51:05.036984Z  INFO static_web_server::info: health endpoint: enabled=false
2024-05-13T16:51:05.036989Z  INFO static_web_server::info: metrics endpoint (experimental): enabled=false
2024-05-13T16:51:05.036995Z  INFO static_web_server::info: basic authentication: enabled=false
2024-05-13T16:51:05.036999Z  INFO static_web_server::info: maintenance mode: enabled=false
2024-05-13T16:51:05.037003Z  INFO static_web_server::info: maintenance mode status: 503
2024-05-13T16:51:05.037007Z  INFO static_web_server::info: maintenance mode file: ""
2024-05-13T16:51:05.037011Z  INFO static_web_server::info: compression static: enabled=false
2024-05-13T16:51:05.037016Z  INFO static_web_server::info: auto compression: enabled=true, formats=deflate,gzip,brotli,zstd
2024-05-13T16:51:05.037022Z  INFO static_web_server::info: cache control headers: enabled=true
2024-05-13T16:51:05.037026Z  INFO static_web_server::info: security headers: enabled=false
2024-05-13T16:51:05.037095Z  INFO static_web_server::info: http1 server is listening on http://[::]:1234
2024-05-13T16:51:05.037106Z  INFO static_web_server::info: press ctrl+c to shut down the server

Ok, exploring a bit more in detail. Turns out that it is a well-known problem with a solution. Basically, we can keep musl static/linked binaries for aarch64 (a.k.a. ARM64) targets but we have to tell jemalloc to use a larger page size by default.

So we can go with using the maximal supported page size of 64KB on aarch64 which should be not a problem for system with smaller page sizes since jemalloc supports that functionality. See https://github.com/jemalloc/jemalloc/pull/769

I will prepare a fix soon.

Thank you @fred00 for testing and @joseluisq for a very fast response

I tried to test the provided binaries outside of kubernetes / container.d to see if they would start. I used ctr to mount the image directly on the host OS. I get the same error.

$ uname -a
Linux XXXX 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux

$ mkdir /tmp/sws

$ sudo ctr i mount docker.io/joseluisq/static-web-server:2.31.0-debian /tmp/sws
sha256:7ec9abf201ddaa3aa824d5711706ee755faf37220e886b2147c16a2bd54287c2
/tmp/sws

$ /tmp/sws/usr/local/bin/static-web-server
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted

$ sudo umount /tmp/sws 

$ sudo ctr i mount docker.io/joseluisq/static-web-server:2.31.0-alpine /tmp/sws
sha256:1c47b434ead451aa2dcfe6ad16a20f0492cb3d60529583ca055087dd007ed321
/tmp/sws

$ /tmp/sws/usr/local/bin/static-web-server
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted

$ sudo umount /tmp/sws 

$ sudo ctr i mount docker.io/joseluisq/static-web-server:2.31.0 /tmp/sws
sha256:9b0d3a7359d63fce6d03d295e2fe2d10e0a1da238ae89147f2457ef3af56a6b2
/tmp/sws

$ /tmp/sws/usr/local/bin/static-web-server
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted

$ sudo umount /tmp/sws 

Thank you! I am really amazed how quickly you responded to this error. The static web server is an amazing software and I really enjoy using it. Thank you for your time and effort, I really appreciate it 😃

Quick Update: I have managed to get a Raspberry Pi 5 at work and it works fine with the binaries that @joseluisq has provided here

Thanks for checking. That’s what I was assuming. I just wanted to double-check.

Now, two things to clarify.

First, regarding the musl static-linked binary for linux/arm64, we still want to compile jemalloc with a 64KB page size default.

Second, regarding the Debian Docker image for linux/arm64, I’m still thinking on why not provide dynamic-linked binaries (glibc) like aarch64-unknown-linux-gnu for the Debian-based images only and leave the static-linked binaries (musl) for the Alpine and the Scratch images for consistency. This has to come up later at some point. But for now, we will fix the issue by increasing the page size as discussed in https://github.com/static-web-server/static-web-server/issues/440#issuecomment-2106958346.

@joseluisq It this how you imagined the Debian test image? Dockerfile

@joseluisq I’m not sure to have the toolchain required to build a new image for aarch64. If @sharknoon can’t do it I will try my best. If you prepare an image with a fix, I will be able to test it with both Pi 5 kernels to provide you with a feedback on the fix compatibility.

Workaround

After some web search, i found a workaround that could confirm the issue with the page 16K page size ( https://github.com/raspberrypi/bookworm-feedback/issues/107 ). I added the kernel=kernel8.img line after the [all] section in /boot/firmware/config.txt file and rebooted the Raspberry Pi 5.

[all]

kernel=kernel8.img

Now uname -a show that the kernel type/version changed from 2712 to 8 :

Linux XXXX 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64

The pods are in the Running state:

NAME                              READY   STATUS    RESTARTS        AGE
sws-deployment-66f5b8b9b8-z9vj8   1/1     Running   111 (16m ago)   9h
sws-deployment-66f5b8b9b8-644jw   1/1     Running   111 (15m ago)   9h

And the logs show a successful start of SWS:

2024-05-12T05:10:05.220487Z  INFO static_web_server::info: static-web-server 2.30.0
2024-05-12T05:10:05.220524Z  INFO static_web_server::info: log level: error
2024-05-12T05:10:05.220596Z  INFO static_web_server::info: server bound to tcp socket [::]:80
2024-05-12T05:10:05.220600Z  INFO static_web_server::info: runtime worker threads: 4
2024-05-12T05:10:05.220603Z  INFO static_web_server::info: runtime max blocking threads: 512
2024-05-12T05:10:05.228592Z  INFO static_web_server::info: log remote address: enabled=false
2024-05-12T05:10:05.228606Z  INFO static_web_server::info: redirect trailing slash: enabled=true
2024-05-12T05:10:05.228611Z  INFO static_web_server::info: ignore hidden files: enabled=false
2024-05-12T05:10:05.228614Z  INFO static_web_server::info: grace period before graceful shutdown: 0s
2024-05-12T05:10:05.228619Z  INFO static_web_server::info: index files: index.html
2024-05-12T05:10:05.228623Z  INFO static_web_server::info: directory listing: enabled=false
2024-05-12T05:10:05.228631Z  INFO static_web_server::info: directory listing order code: 6
2024-05-12T05:10:05.228634Z  INFO static_web_server::info: directory listing format: Html
2024-05-12T05:10:05.237242Z  INFO static_web_server::info: fallback page: enabled=false, value=""
2024-05-12T05:10:05.237256Z  INFO static_web_server::info: health endpoint: enabled=false
2024-05-12T05:10:05.237259Z  INFO static_web_server::info: metrics endpoint (experimental): enabled=false
2024-05-12T05:10:05.237266Z  INFO static_web_server::info: basic authentication: enabled=false
2024-05-12T05:10:05.237269Z  INFO static_web_server::info: maintenance mode: enabled=false
2024-05-12T05:10:05.237272Z  INFO static_web_server::info: maintenance mode status: 503
2024-05-12T05:10:05.237275Z  INFO static_web_server::info: maintenance mode file: ""
2024-05-12T05:10:05.237278Z  INFO static_web_server::info: compression static: enabled=false
2024-05-12T05:10:05.237283Z  INFO static_web_server::info: auto compression: enabled=true, formats=deflate,gzip,brotli,zstd
2024-05-12T05:10:05.237286Z  INFO static_web_server::info: cache control headers: enabled=true
2024-05-12T05:10:05.237289Z  INFO static_web_server::info: security headers: enabled=false
2024-05-12T05:10:05.252081Z  INFO static_web_server::info: http1 server is listening on http://[::]:80
2024-05-12T05:10:05.252091Z  INFO static_web_server::info: press ctrl+c to shut down the server