moby: Documentation: Make clear instructions for getting a core file, when container crashes

@sdivekar I’d recommending setting the core_pattern at the docker host level. The pattern is a kernel-level parameter so I suspect if multiple containers try to set it to different things you’ll end up with a last-write-wins case where some containers won’t be able to dump a core file. Not sure if you saw my comment here but setting the pattern at the host level also avoids the need for running in privileged mode.

I’d suggest that all of this works around the problem. Reducing the confinement of the application, sometimes significantly in order to manage the core isn’t the solution that I see people wanting. Since Docker is orchestrating the container, the preference would be for Docker to configure the location and methods for the kernel to deliver the core file. Perhaps this could be inside of the container, or in a related location (outside of the container root, but still managed space such as a volume), or in the system wide location.

• It is not clear, as core dumps are a kernel managed thing, whether resulting core files are written by kernel to the host filesystem or the container filesystem. This should also be clearly documented. They are written inside the container.

I haven’t seen this mentioned in this thread, but the truth is a bit more complicated.

If /proc/sys/kernel/core_pattern refers to a fixed path, it is indeed resolved to a path in the container’s namespace.

However, if /proc/sys/kernel/core_pattern is set to pipe into a script/binary, it is resolved to path on the host’s filesystem. This also means that the core dump will likely end up being stored on the host itself.

The best documentation I’ve found for this oddity so far is this email thread: https://lkml.org/lkml/2015/10/24/134

I would contribute my findings, which certainly overlap with the above said, but might contain complementary bits:

• /proc/sys/core_pattern is clearly persisted on the host. Taking note of it’s content before starting any endeavour is therefore highly encouraged.
• dockers --privileged is necessary for a gdb session to catch the stack, without privileges, gdb just complains about No stack. Google still is hardly knowledgeable about this phenomenon…
• setting ulimit on docker run works perfectly, for future googlers (syntax hard to find), an docker-compose example:

ulimits:
      core: -1

By using --privileged=true and modifying /proc/sys/kernel/core_pattern inside the container, you are actually modifying the host system’s /proc/sys/kernel/core_pattern, so you might as well just modify it on the host to begin with.

Just thought I’d share one more discovery on this front.

In order to test process crashes I tried using kill -ABRT but apparently you can’t kill -ABRT 1.

Killing the main PID Of the container with SIGABRT seems to be ignored both from the host or inside a docker exec call.

My work around is to run the process under sh -c. Then ABRT on the sub-process works as expected.

More problems:

When I set the core_pattern on host

echo '/config/core.%h.%e.%t' > /proc/sys/kernel/core_pattern
echo '/recordings/core.%h.%e.%t' > /proc/sys/kernel/core_pattern

… no core file is created. Yet if I set the path to be /tmp then it magically works ?!!?!??

echo '/tmp/core.%h.%e.%t' > /proc/sys/kernel/core_pattern

I don’t understand.

Other problem:

Setting shebang to #!/bin/bash and it still didn’t set the ulimit

[EDIT] I’ve since been told that setting the ulimit is only applied for the context (scope) and duration of the script. So the setting is not seen from other launched processes inside the container (e.g. docker exec -it).

The core file is created. Into /tmp only. Nothing else so far has worked for me. It would be better if we could configure the ulimit -c unlimited setting at build time for the whole container. Rather than at run time and only for certain processes.

Please someone else check these things out. Confirm / deny.

@praveenmak When running processes in containers (or, generally speaking, with PID namespaces), processes will have one PID per namespace that they belong to; and namespaces are nested. If that feels confusing, you can try the following commands, on a Linux machine:

CONTAINERID=$(docker run -d nginx)
docker exec $CONTAINERID ps faux
ps faux | grep nginx

You will see that these NGINX processes have a PID inside the container, and a PID outside the container. So that corresponds to %p and %P respectively. I hope that helps!

Hi,

The handle_core.sh script can find the path of the container by using the %P variable (PID of the crashed process, seen by the host) + /proc/<PID>/ns. This can be simplified by using nsenter to then enter the mnt namespace.

For instance, the following will create the core file in /core in the container, and it will create a little log file (for troubleshooting issues) in /tmp/core.<PID> on the host as well:

nsenter -t $1 -m tee /core >/dev/null 2>>/tmp/core.$$

However, this only works if the namespaces still exists at dump time. If the crashed process causes the whole container to exist, then the namespaces won’t exist anymore, unfortunately. From what I understand, this is because Linux correctly waits for the core handler to run, but if the whole namespaces gets destroyed for another reason (e.g. PID 1 in the container exits), then you’re out of luck.

I hope this helps!

@bcantrill ok Brian. A working example where you can see it in action / take thing from is my tvh.debug docker image:

https://registry.hub.docker.com/u/dreamcat4/tvh.debug/dockerfile/

Script: debug.sh

https://github.com/dreamcat4/docker-images/blob/master/tvh/debug/stable/debug.sh

Usage:

https://github.com/dreamcat4/docker-images/blob/master/tvh/README.md#debugging-tvheadend

Caveat:

/crash must be bind-mounted as a host volume. It doesn’t work otherwise.

• Must be run in priveleged mode.