docker-nginx: Segfault during requests with 1.25.2 - LUA issue

Environment

  • most recent image (nginx 1.25.2) from 17 Oct
  • using nginx:mainline-bookworm

Bug

When building our image including ndk/lua/brotli with 1.25.2 as baseline, most of our XHR calls return an empty response with an crash / coredump.

Details

To be more of a help, i tried to get the core dump. I have it, but i miss the symbols (starting with nginx-debug). I could not find any hints/docs how it is intended to get the symbols for gdb backtrace so i could share any more useful information here.

Can someone hint me to the docs or explain how to get the symbols (without recompiling nginx myself)?

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Comments: 39

Most upvoted comments

If you are in mood and want to add something to the docs on how to get core dumps up and running, those are the complete docs - happy to share those.


Setting up for core-dump stacktrace support

On you host running docker

# enable coredumps
echo "/tmp/cores/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern
sudo sysctl -w fs.suid_dumpable=2
ulimit -c unlimited

On the container

vi /etc/nginx/nginx.conf

# add this at the very top
worker_rlimit_core  1000M;
working_directory /tmp/cores;

# install headers and gdb
export NGINX_GPGKEY_PATH=/usr/share/keyrings/nginx-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] https://nginx.org/packages/mainline/debian/ bookworm nginx" >> /etc/apt/sources.list.d/nginx.list
apt update && apt install -y nginx-dbg nginx-module-njs-dbg nginx-module-njs-dbg libc6-dbg gdb

mkdir -p /tmp/cores
chown root:root /tmp/cores
chmod 1777 /tmp/cores

Now on the container, start nginx-debug

nginx-debug

Reproduce your crash and then open the core-dump with (replace the XXXX)

gdb --core=/tmp/cores/core.nginx-debug.XXXX /usr/sbin/nginx-debug

Specials / Optional for custom modules

If you are using addition/custom modules, you need to adjust the installation of those to include the symbols too, so if those symbols show up in your stackrace, they can be resolved too. For this add this below this line https://github.com/nginxinc/docker-nginx/blob/master/modules/Dockerfile#L78

apt install --no-install-suggests --no-install-recommends -y /tmp/packages/nginx-module-${module}-dbg_${NGINX_VERSION}*.deb; \

Hey @EugenMayer - thanks for checking in. The current ETA is mid-January I think, but it can slip a couple weeks further too.

There is no ETA as of now…

Thanks for the tests!

So what i have now is

nginx -version
nginx version: nginx/1.25.3

# installed your .deb packages for ndk / lua

dpkg --list | grep nginx
ii  nginx                       1.25.3-1~bookworm              amd64        high performance web server
ii  nginx-module-brotli         1.25.3+1.0.0-1~bookworm        amd64        nginx 3rd-party brotli compression dynamic modules
ii  nginx-module-brotli-dbg     1.25.3+1.0.0-1~bookworm        amd64        debug symbols for the nginx-module-brotli
ii  nginx-module-geoip          1.25.3-1~bookworm              amd64        nginx GeoIP dynamic modules
ii  nginx-module-image-filter   1.25.3-1~bookworm              amd64        nginx image filter dynamic module
ii  nginx-module-lua            1.25.3+0.10.25-1~bookworm      amd64        nginx 3rd-party Lua dynamic modules
ii  nginx-module-ndk            1.25.3+0.3.2-1~bookworm        amd64        nginx 3rd-party NDK dynamic module
ii  nginx-module-njs            1.25.3+0.8.2-1~bookworm        amd64        nginx njs dynamic modules
ii  nginx-module-xslt           1.25.3-1~bookworm              amd64        nginx xslt dynamic module

and it works! Using the original lua/ndk modules (not your patched ones), i get empty responses with the segfault all over the place.

So indeed, your build fixes it!

Correct. Well, provided I’m right about the patch and if it will fix the issue for you (but if it works with 1.25.1 this should be true).