envoy: Envoy 1.17-1.23 unable to start on Raspberry

Envoy crashes on RP 4 due to `tcmalloc` allocation error

I am trying to run Envoy on RP4 and 64bit Raspberry Pi OS.

Envoy 1.20 crashes with:

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:550] MmapAligned() failed (size, alignment) 1073741824 1073741824 @ 0x559342edc4 0x55934211a4 0x5593420be4 0x559340a4c0
external/com_github_google_tcmalloc/tcmalloc/arena.cc:34] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size) 131072 48 @ 0x559342f0d4 0x559340a534

Envoy 1.23 crashes with:

external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:631] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x151880000000 1073741824 1073741824 @ 0x55941cfa88 0x55941cb6a0 0x55941caf48 0x55941b2f4c 0x55941c8010 0x55941c7e2c 0x55941a78a8 0x55940e6c00 0x55940e2298 0x559406f484 0x559419dbd0 0x7f834774ac
external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 600 @ 0x55941cfde4 0x55941b2fdc 0x55941c8010 0x55941c7e2c 0x55941a78a8 0x55940e6c00 0x55940e2298 0x559406f484 0x559419dbd0 0x7f834774ac

Tested also on envoy 1.17, 1.18, 1.19, and 1.22 (it seems that tcmalloc was introduced in 1.17). Same behaviour.

How to reproduce:

In Docker:

docker run --rm envoyproxy/envoy:v1.24-latest

On a host machine:

envoy_location=$(docker run --rm --entrypoint which docker.io/envoyproxy/envoy:v1.23-latest envoy)
container_id=$(docker create which envoyproxy/envoy:v1.23-latest)
docker cp "${container_id}":"${envoy_location}" ./envoy
docker rm -v "${container_id}"
./envoy

I am using RP4 8GB.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 19 (6 by maintainers)

Links to this issue

Most upvoted comments

Or compiling envoy with --define tcmalloc=gperftools which for me was the only acceptable option. I have published the images on Docker Hub https://hub.docker.com/repository/docker/thegrandpkizzle/envoy if you don’t want to compile it yourself.

PKizzle on Oct 25, 2022

Just for sake of completeness, you can use a custom-compiled Raspberry Kernel with having CONFIG_PGTABLE_LEVELS changed:

# working when running as root
# install build requirements
apt install -y git bc bison flex libssl-dev make libncurses5-dev
# clone and download kernel sources
cd /opt
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux

# create kernel config
#RASPI 4: `make bcm2711_defconfig`
#RASPI 5: `make bcm2712_defconfig`

# # DO OTHER TUNINGS HERE via `make menuconfig`

# set PG table size to 4 instead of default 3
sed -i 's/CONFIG_PGTABLE_LEVELS=3/CONFIG_PGTABLE_LEVELS=4/' .config

# compile (requires a lot of time on raspi 4)
make -j4 Image.gz modules dtbs
make modules_install

# copy compiled resources
cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
cp arch/arm64/boot/dts/overlays/README /boot/overlays/
# RASPI 4: cp arch/arm64/boot/Image.gz /boot/kernel8.img
# RASPI 5: cp arch/arm64/boot/Image.gz /boot/firmware/kernel8.img

# use new kernel
echo "kernel=kernel8.img" | tee -a /boot/config.txt

reboot

This should make it working without a custom compiled envoy. Sadly the Raspberry issue https://github.com/raspberrypi/linux/issues/4375 (which is already mentioned in this thread https://github.com/envoyproxy/envoy/issues/23339#issuecomment-1267200594) is still open

I am running Hashicorp Nomad with that custom compiled kernel (with some other tweaks), hopefully works for others too.

FibreFoX on Mar 10, 2024

seems like there is some resistance to switching the required kernel flags in raspberryPI OS - workarounds seem to be switching to debian or ubuntu or recompiling the kernel

phlax on Oct 4, 2022

I’ll try to build with --define tcmalloc=disabled (I’ll try containerised approach) and see whether it will work.

Sitin on Oct 1, 2022

envoy: Envoy 1.17-1.23 unable to start on Raspberry

Envoy crashes on RP 4 due to tcmalloc allocation error

About this issue

Links to this issue

Most upvoted comments

Envoy crashes on RP 4 due to `tcmalloc` allocation error