moby: Kill `docker exec` command will not terminate the spawned process

Yo, 2017! Any love?

Is there any movement on this? We’ve just hit this issue as well - very unexpected behavior.

This issue was the root cause of a number of inconveniences I’ve experienced over the past several months, and only today did I finally land on this bug.

The workaround isn’t too fun or easy either. I’m using Python to call docker exec in a subprocess, and what I settled on amounts to grepping docker exec ps ... to get the PID of the command I just ran, followed by docker exec kill ... to kill the process running inside the container. There were also some tricky aspects to what I had to do, but I won’t describe them here.

I think this issue should be prioritized more highly because it’s the kind of behavior one takes for granted, and in certain use cases (like in my case) it’s easy not to notice this bug was happening all along.

This issue is still very much existing and causing unexpected and unwanted behavior.

I had that issue with both docker run and docker exec not dispatching signals to the daemon/container. The root cause is when using --tty signal proxying is entirely disabled with no way to enable it (even with --sig-proxy). It affects at least docker run and docker exec that share the same code path.

Previously --sig-proxy was an option to force proxying signals when not using a tty, and hence when using --tty the proxy got forwarded. Passing both --sig-proxy and -tty was leading to an error. That looked like:

Options	Signal proxy?
none	No
--tty	Yes
--sig-proxy	Yes
--tty --sig-proxy	error: TTY mode (-t) already imply signal proxying (-sig-proxy)

October 2013 patch e0b59ab52b87b8fc15dd5534c3231fdd74843f9f was made to “_Enable sig-proxy by default in run and attach _”. It changed --sig-proxy to default to true and made --tty to always disable signal proxy.

Options	Signal proxy?
none	Yes (was No)
--tty	No (was Yes)
--sig-proxy	Yes
--tty --sig-proxy	No (was an error)

So what happened with e0b59ab52b87b8fc15dd5534c3231fdd74843f9f is that signal proxying is now enabled by default for non-tty. BUT the patch has a fault: setting --tty always disable signal proxying.

I am pretty sure that signal-proxying should be enabled by default whether it is a tty or non-tty mode, it is still possible to disable it with --sig-proxy=false. A patch would thus have to implement the following changes:

Options	Signal proxy (current)	Expected
none	Yes	Yes
--tty	No	Yes
--sig-proxy	Yes	Yes
--sig-proxy=false	No	No
--tty --sig-proxy	No	Yes
--tty --sig-proxy=false	No	No

TLDR: --tty should not arbitrarily force sigProxy = false caused by e0b59ab52b87b8fc15dd5534c3231fdd74843f9f

Reference: https://phabricator.wikimedia.org/T176747#3749436

Greetings from the year 2022, where I lost my Friday night to this issue. I was also in the situation where I was trying to run docker exec commands from within a python subprocess.Popen call. My most elegant workaround was to find the PID of the persistent process inside the docker container by using subprocess.run to execute:

docker exec my_container bash -c 'pgrep -xf "my_exact_persistent_command"'

with that PID, I’m able to easily kill it with another subprocess.run of:

docker exec my_container bash -c 'kill my_pid'

Hello. I’ve run into this issue too. I offer here a workaround done to the best of my limited understanding of how docker works and its internals (most learned while creating this workaround).

First, let me explain the use case: I run a docker image which runs a basic piece of software (a driver, long-time running process, using a number of random ports). Over time, I need to run further commands to interact with this driver, these commands are also long-time running processes. They interact with the driver and in doing so they need to get some resources and free them properly afterwards, which implies, they need to be killed cleanly, i.e. when getting SIGTERM or SIGINT they must do cleanup before exiting. All this software is automatically run by a tty-less environment (supervisord). Summary: I do docker run -t FLAGS --name DRIVER_CONTAINER MYIMAGE MYDRIVER and later on every X time docker exec -t DRIVER_CONTAINER MYDRIVERCLIENT. I need the exec commands to exit cleanly and not remain as zombies.

Now, the fun part, my workaround: I made a wrapper, using the Docker API and some hacky bits, for docker exec to enable signal forwarding. It is contained in this gist which has the class DockerExecWithSignalForwarding. It contains a command line interface that imitates docker exec, supporting (almost) the same flags and nature of use. I install it as dockerexec to make it look similar to docker exec. It supports remote DOCKER_HOST via SSH.

I hope this may help by: giving an option to people falling into the same issue, and maybe providing an idea on how to do it officially (if there is no better way). And I’d also request feedback on how to improve it (in the comments of the gist probably).

I reproduce it here as it is right now for ease of reading:

#!/usr/bin/env python3
import sys
import argparse
import os
import subprocess
import signal
import time
import threading
import shlex
import docker

"""
This program is meant to substitute running 'docker exec <FLAGS> CONTAINER_NAME COMMAND'
to overcome the limitation of docker exec not forwarding signals to the
executed process.

This was reported here in Nov 2014: https://github.com/moby/moby/issues/9098#issuecomment-312152980)
Furthermore, here: https://github.com/docker/cli/pull/1841 they say

    moby/moby#9098 Kill docker exec command will not terminate the spawned process
        This patch does not fix the docker exec case; it looks like there's no API to kill an exec'd process, 
        so there's no signal-proxy for this yet
        
Note: -i is not supported, couldn't make it work but got as close as I could (maybe it can't work?).

Author: Sammy Pfeiffer <sam.pfeiffer at hullbot.com>
"""


class DockerExecWithSignalForwarding(object):
    def __init__(self, container_name, command,
                 # Offer the rest of the options from docker exec
                 detach=False,
                 # Note detach-keys is not implemented
                 environment=None,
                 # Not supported
                 interactive=False,
                 privileged=False,
                 tty=False,
                 # Leaving these as None makes the inner docker API deal with them correctly
                 user=None,
                 workdir=None,
                 # By default the timeout of Python's docker exec is 60s, change it to 1 year-ish
                 socket_timeout=60 * 60 * 24 * 365):
        """
        Provided a set of flags (same ones of docker exec), a container name and a command,
        do 'docker exec' but managing signals to be forwarded to the exec-ed process.
        """
        if interactive:
            raise RuntimeError("Interactive mode not supported, use docker exec.")
        # We inherit the rest of the configuration (including DOCKER_HOST) from the environment
        self.client = docker.from_env(timeout=socket_timeout)

        # Sanity check on the command, should be a string which we split with shlex or already a list/tuple
        if isinstance(command, str):
            command = shlex.split(command)
        if not (isinstance(command, list) or isinstance(command, tuple)):
            raise TypeError("Command is of type {} and it must be str/list/tuple. (command: {})".format(
                type(command),
                command))

        # Translate docker exec style arguments into exec_run arguments
        try:
            # Get a reference to the container
            self.container = self.client.containers.get(container_name)
            # Get the Id of the 'docker exec' instance (that is not yet being executed) so we can start it
            exec_create_response = self.client.api.exec_create(self.container.id,
                                                               command,
                                                               stdout=True,
                                                               stderr=True,
                                                               stdin=self.interactive,
                                                               tty=tty,
                                                               privileged=privileged,
                                                               user=user,
                                                               environment=environment,
                                                               workdir=workdir)
            self.exec_id = exec_create_response['Id']

            # The following block of code is to manage the situation of an interactive session
            # We would like to support it but the underlying API doesn't allow for it (writing into the socket
            # simply does not work as far as I could test) it was a lot of work to figure out the bits
            # to get this to this state, so I'm leaving it here
            # if interactive:
            #     # Because we want to support stdin we need to access the lower-level socket
            #     # instead of being able to use exec_start with stream=True
            #     self.exec_socket = self.client.api.exec_start(self.exec_id,
            #                                                   detach=detach,
            #                                                   tty=tty,
            #                                                   stream=False,
            #                                                   socket=True,
            #                                                   demux=True)

            #     # Recreate the function that offers the generator for output usually when using stream=True
            #     def _read_from_socket(socket, stream, tty=True, demux=False):
            #         """
            #         Adapted from docker/client.py in order to enable stdin... tricky.
            #         """
            #         gen = docker.api.client.frames_iter(socket, tty)

            #         if demux:
            #             # The generator will output tuples (stdout, stderr)
            #             gen = (docker.api.client.demux_adaptor(*frame) for frame in gen)
            #         else:
            #             # The generator will output strings
            #             gen = (data for (_, data) in gen)

            #         if stream:
            #             return gen
            #         else:
            #             # Wait for all the frames, concatenate them, and return the result
            #             return docker.api.client.consume_socket_output(gen, demux=demux)

            #     self.exec_output = _read_from_socket(self.exec_socket, True, tty, True)
            # else:
            self.exec_output = self.client.api.exec_start(self.exec_id,
                                                          detach=detach,
                                                          tty=tty,
                                                          stream=True,
                                                          socket=False,
                                                          demux=True)

            self.setup_signal_forwarding()
            self.program_running = True

        # Imitate the behaviour of the original docker exec up to a point
        except docker.errors.NotFound as e:
            print("Error: No such container: {}".format(container_name))
            os._exit(1)

        # Start a thread that monitors if the program died so we can end this when this happens
        self.monitor_thread = threading.Thread(target=self.monitor_exec)
        self.monitor_thread.start()

        self.output_manager_thread = None
        if self.interactive:
            # Deal with stdout and stderr in a thread and let the main thread deal with input
            self.output_manager_thread = threading.Thread(target=self.manage_stdout_and_stderr)
            self.output_manager_thread.start()
            self.manage_stdin()
        else:
            self.manage_stdout_and_stderr()

    def monitor_exec(self):
        """
        We loop (very slowly) to check if the underlaying command died, this is useful for
        commands executed in a remote docker daemon. It 'should' not happen locally, but it may.
        """
        try:
            # Check if the process is dead, the 'Running' key must become false
            exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
            while exec_inspect_dict.get('Running'):
                # Generous sleep, as this is to catch the program dieing by something else than this wrapper
                time.sleep(10.0)
                exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)

            # If it's dead, we should exit with its exit code
            os._exit(exec_inspect_dict.get('ExitCode'))

        except docker.errors.APIError as e:
            # API error, we can't access anymore, exit
            raise RuntimeError("Docker API error when monitoring exec process ({})".format(e))

    def forward_signal(self, signal_number, frame):
        """
        Forward the signal signal_number to the container,
        we first need to find what's the in-container PID of the process we docker exec-ed
        then we docker exec a kill signal with it.
        """
        # print("Forwarding signal {}".format(signal_number))
        # Using a lock to attempt to deal with Control+C spam
        with self.signal_lock:
            pid_in_container = self.get_container_pid()
            kill_command = ["kill", "-{}".format(signal_number), str(pid_in_container)]
            try:
                exit_code, output = self.container.exec_run(kill_command,
                                                            # Do it always as root
                                                            user='root')
            except docker.errors.NotFound as e:
                raise RuntimeError("Container doesn't exist, can't forward signal {} (Exception: {})".format(
                    signal_number, e))

            if exit_code != 0:
                raise RuntimeError(
                    'When forwarding signal {}, kill command to PID in container {} failed with exit code {}, output was: {}'.format(
                        signal_number, pid_in_container, exit_code, output))

    def get_container_pid(self):
        """
        Return the in-container PID of the exec-ed process.
        """
        try:
             # I wish the stored PID of exec was the container PID (which is what I expected)
             # but it's actually the host PID so in the following lines we deal with it
            pid_in_host = self.client.api.exec_inspect(self.exec_id).get('Pid')
        except docker.errors.NotFound as e:
            raise RuntimeError("Container doesn't exist, can't get exec PID (Exception: {})".format(e))

        # We need to translate the host PID into the container PID, there is no general mapping for it in Docker
        # If we are running in the same host, this is easier, we can get the Docker PID by just doing:
        #       cat /proc/PID/status | grep NSpid | awk '{print $3}'
        # If the docker container is running in a different machine we need to execute that command in that machine
        # which implies using SSH to execute the command

        # Here we can only support DOCKER_HOST=ssh://user@host to use ssh to execute this command
        # as if we are using ssh:// to access the docker daemon it's fair to assume we have SSH keys setup
        # if docker host is tcp:// on another host or a socket file with SSH tunneling there isn't much we can do
        docker_host = os.environ.get('DOCKER_HOST', None)
        # If using SSH execute the command remotely
        if docker_host and 'ssh://' in docker_host:
            ssh_user_at_host = docker_host.replace('ssh://', '')
            get_pid_in_container_cmd = "ssh -q -o StrictHostKeyChecking=no {} ".format(ssh_user_at_host)
            get_pid_in_container_cmd += "cat /proc/{}/status | grep NSpid | awk '{{print $3}}'".format(pid_in_host)
        # Otherwise, execute the command locally
        else:
            get_pid_in_container_cmd = "cat /proc/{}/status | grep NSpid | awk '{{print $3}}'".format(pid_in_host)

        # Execute the command that gets the in-Docker PID
        try:
            pid_in_container = subprocess.check_output(get_pid_in_container_cmd, shell=True)
        except subprocess.CalledProcessError as e:
            raise RuntimeError(
                "CalledProcessError exception while trying to get the in-docker PID of the process ({})".format(e))

        return int(pid_in_container)

    def setup_signal_forwarding(self):
        """
        Forward all signals to the docker exec-ed process.
        If it dies, this process will die too as self.manage_stdout_and_stderr will finish
        and forward the exit code.
        """
        self.signal_lock = threading.Lock()
        # Forward all signals, even though we are most interested just in SIGTERM and SIGINT
        signal.signal(signal.SIGHUP, self.forward_signal)
        signal.signal(signal.SIGINT, self.forward_signal)
        signal.signal(signal.SIGQUIT, self.forward_signal)
        signal.signal(signal.SIGILL, self.forward_signal)
        signal.signal(signal.SIGTRAP, self.forward_signal)
        signal.signal(signal.SIGABRT, self.forward_signal)
        signal.signal(signal.SIGBUS, self.forward_signal)
        signal.signal(signal.SIGFPE, self.forward_signal)
        # Can't be captured, but for clarity leaving it here
        # signal.signal(signal.SIGKILL, self.forward_signal)
        signal.signal(signal.SIGUSR1, self.forward_signal)
        signal.signal(signal.SIGUSR2, self.forward_signal)
        signal.signal(signal.SIGSEGV, self.forward_signal)
        signal.signal(signal.SIGPIPE, self.forward_signal)
        signal.signal(signal.SIGALRM, self.forward_signal)
        signal.signal(signal.SIGTERM, self.forward_signal)

    def manage_stdout_and_stderr(self):
        """
        Print stdout and stderr as the generator provides it.
        When the generator finishes we exit the program forwarding the exit code.
        """
        # Note that if the application prints a lot, this will use some CPU
        # but there is no way around it as we are forced to read from the socket and decode to print
        for stdout, stderr in self.exec_output:
            # Note that if choosing tty=True output is always in stdout
            if stdout:
                print(stdout.decode("utf-8"), file=sys.stdout, end='')
            if stderr:
                print(stderr.decode("utf-8"), file=sys.stderr, end='')

        # When we come out of this loop, the program we exec-ed has terminated
        # so we can exit with its exit code just here
        exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
        exit_code = exec_inspect_dict.get('ExitCode')
        os._exit(exit_code)

    def manage_stdin(self):
        """
        Forward the input of this program to the docker exec-ed program.
        """
        raise NotImplemented("Managing stdin is not implemented.")
        # print(dir(self.exec_socket))
        # print(self.exec_socket.readable())
        # print(self.exec_socket.writable())
        # print(dir(self.exec_socket._sock))
        # self.exec_socket._writing = True
        # print(self.exec_socket.writable())
        # def write(sock, str):
        #     while len(str) > 0:
        #         written = sock.write(str)
        #         str = str[written:]
        # while True:
        #     # self.exec_socket._sock.sendall(input().encode('utf-8'))
        #     # self.exec_socket.flush()
        #     #print("sent")
        #     # Doesn't work either
        #     write(self.exec_socket, input().encode('utf-8'))
        #     print("--written--")
        #     #os.write(self.exec_socket._sock.fileno(), input().encode('utf-8'))
        #     #print("sent")
        #     #print("Received: {}".format(self.exec_socket._sock.recv(1)))
        #     # try:
        #     #     print(os.read(self.exec_socket._sock.fileno(), 4096))
        #     # except BlockingIOError as b:
        #     #     print("BlockingIOError: {} ".format(b))
        #     #     print(self.client.api.exec_inspect(self.exec_id))

    def __del__(self):
        """
        When the program ends this gets called so we can cleanup resources
        and exit with the exit code from the exec-ed command.
        Note it is unlikely this gets ever called.
        """
        # print("Calling __del__")
        # Wait for the output thread in case there are more prints to show
        if self.output_manager_thread:
            self.output_manager_thread.join()

        # Try to wait for the process to be dead in case it isn't yet
        try:
            exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
            while exec_inspect_dict.get('Running'):
                time.sleep(0.1)
                exec_inspect_dict = self.client.api.exec_inspect(self.exec_id)
        except docker.errors.APIError as e:
            # We may get an API error here, if so, return an exit code other than 0
            os._exit(127)
            pass

        # Forward the exit code of the exec-ed command if we got here
        exit_code = exec_inspect_dict.get('ExitCode')
        os._exit(exit_code)


if __name__ == '__main__':
    # Original docker exec --help
    """
Usage:	docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

Run a command in a running container

Options:
  -d, --detach               Detached mode: run command in the background
      --detach-keys string   Override the key sequence for detaching a container
  -e, --env list             Set environment variables
  -i, --interactive          Keep STDIN open even if not attached
      --privileged           Give extended privileges to the command
  -t, --tty                  Allocate a pseudo-TTY
  -u, --user string          Username or UID (format: <name|uid>[:<group|gid>])
  -w, --workdir string       Working directory inside the container
"""
    parser = argparse.ArgumentParser(description="Run a command in a running container")
    parser.add_argument("container", help="Container name")
    parser.add_argument("command_and_args", help="Command and arguments", nargs=argparse.REMAINDER)

    parser.add_argument("-d", "--detach", action='store_true',
                        help="Detached mode: run command in the background")
    # We only support environment variables as a long string if there must be more than one
    # I.e. -e USER=user for one or -e "USER=user SOMETHING_ELSE=1"
    # Supporting multiple -e didn't work for me
    parser.add_argument("-e", "--env",
                        type=str, help="Set environment variables (like 'VAR1=1 VAR2=2')")
    # Interactive is not supported, but leaving it here just in case it is implemented in the future
    parser.add_argument("-i", "--interactive", action='store_true',
                        help="Keep STDIN open even if not attached (Note: not implemented, use 'docker exec')")
    parser.add_argument("--privileged", action='store_true',
                        help="Give extended privileges to the command")
    parser.add_argument("-t", "--tty", action='store_true',
                        help="Allocate a pseudo-TTY")
    parser.add_argument("-u", "--user",
                        type=str, help="Username or UID (format: <name|uid>[:<group|gid>])")
    parser.add_argument("-w", "--workdir",
                        type=str, help="Working directory inside the container")

    args = parser.parse_args()

    if len(args.command_and_args) < 1:
        print("dockerexec requires at least 2 arguments")
        parser.print_help()
        exit(1)

    if args.interactive:
        raise NotImplemented("Interactive mode not implemented, you should just use docker exec")

    dewsf = DockerExecWithSignalForwarding(args.container,
                                           args.command_and_args,
                                           detach=args.detach,
                                           # Note detach-keys is not implemented
                                           environment=args.env,
                                           interactive=args.interactive,
                                           privileged=args.privileged,
                                           tty=args.tty,
                                           user=args.user,
                                           workdir=args.workdir)

    # The following lines are tests done with a container running:
    # docker run --rm -t --name exec_signal_problem python:3 sleep 999
    # Proper testing should be implemented based on this
    # # Forward error test
    # de = DockerExec('exec_signal_problem',
    #                 'ls asdf',
    #                 tty=True,
    #                 interactive=False)
    # # simple working test
    # de = DockerExec('exec_signal_problem',
    #                 'ls',
    #                 tty=True,
    #                 interactive=False)
    # Test signal forwarding SIGINT Control C
    # de = DockerExec('exec_signal_problem',
    #                 'python -c "import sys;import signal;signal.signal(signal.SIGINT, print);print(\'hello\', file=sys.stderr);import time; time.sleep(600)"',
    #                 tty=True,
    #                 interactive=False)
    # Test signal forwarding SIGTERM
    # de = DockerExec('exec_signal_problem',
    #                 'python -c "import sys;import signal;signal.signal(signal.SIGTERM, print);print(\'hello\', file=sys.stderr);import time; time.sleep(600)"',
    #                 tty=True,
    #                 interactive=False)

    # Test output in stderr
    # de = DockerExec('exec_signal_problem',
    #                 'python -c "import sys; print(\'hello stderr\', file=sys.stderr);print(\'hello stdout\', file=sys.stdout)"',
    #                 tty=False,
    #                 interactive=False)

    # test input, doesn't work, not supported (not needed anyways)
    # de = DockerExec('exec_signal_problem',
    #                 'cat',
    #                 tty=True,
    #                 interactive=True)

The following bash snippet can be used as a workaround for this issue. I basically intercept the SIGTERM to docker exec and do a manual cleanup. It is based on this: http://veithen.github.io/2014/11/16/sigterm-propagation.html

function docker_cleanup {
    docker exec $IMAGE bash -c "if [ -f $PIDFILE ]; then kill -TERM -\$(cat $PIDFILE); rm $PIDFILE; fi"
}

function docker_exec {
    IMAGE=$1
    PIDFILE=/tmp/docker-exec-$$
    shift
    trap 'kill $PID; docker_cleanup $IMAGE $PIDFILE' TERM INT
    docker exec $IMAGE bash -c "echo \"\$\$\" > $PIDFILE; exec $*" &
    PID=$!
    wait $PID
    trap - TERM INT
    wait $PID
}

#use it like this:
docker_exec container command arg1 ...

@vishh do you think adding support for POST /exec/:name/stop (and maybe POST /exec/:name/kill) make senses here ( similar to POST /containers/:name/stop and POST /containers/:name/kill ) ? That would actually solve majority of my usecase as I mainly consume the remote API ( which makes the exec process’s unique id available with POST /exec/:name/create )

It’s probably much harder to do it from the docker cli though as we don’t really expose the exec’s id anywhere.

+1

@th3mis no one is saying this is an issue with docker run. It’s an issue with docker exec and your example still doesn’t work. Signals are not forwarded when you kill the docker exec process no matter what combination of commands you pass to it.

$ docker run --init -di ubuntu:16.04
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
21d2b053d10f        ubuntu:16.04        "/bin/bash"         3 minutes ago       Up 3 minutes                            reverent_perlman
$ ps -eaf --forest
root      1846     1  0 Mar06 ?        00:30:23 /usr/bin/dockerd -H fd://
root      8823  1846  0 Mar06 ?        00:16:40  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir 
root       563  8823  0 10:38 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root       583   563  0 10:38 ?        00:00:00          \_ /dev/init -- /bin/bash
root       623   583  0 10:38 ?        00:00:00              \_ /bin/bash

$ docker exec reverent_perlman bash -c '/dev/init -s -- sleep 7777'
<open a new terminal>
$ ps -eaf --forest
root      1846     1  0 Mar06 ?        00:30:24 /usr/bin/dockerd -H fd://
root      8823  1846  0 Mar06 ?        00:16:41  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir 
root       563  8823  0 10:38 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root       583   563  0 10:38 ?        00:00:00      |   \_ /dev/init -- /bin/bash
root       623   583  0 10:38 ?        00:00:00      |       \_ /bin/bash
root      1404  8823  0 10:43 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root      1454  1404  0 10:43 ?        00:00:00          \_ /dev/init -s -- sleep 7777
root      1460  1454  0 10:43 ?        00:00:00              \_ sleep 7777

$ ps -eaf | grep 'docker exec'
root     1715 32646  0 10:45 pts/18   00:00:00 docker exec reverent_perlman bash -c /dev/init -s -- sleep 7777
$ kill 1715
<docker exec in original terminal has exited>
$ ps -eaf --forest
root      1846     1  0 Mar06 ?        00:30:25 /usr/bin/dockerd -H fd://
root      8823  1846  0 Mar06 ?        00:16:41  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir 
root       563  8823  0 10:38 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root       583   563  0 10:38 ?        00:00:00      |   \_ /dev/init -- /bin/bash
root       623   583  0 10:38 ?        00:00:00      |       \_ /bin/bash
root      1944  8823  0 10:47 ?        00:00:00      \_ docker-containerd-shim 21d2b053d10f9d26e294ff055494b4136624fae1faa0952ba63e6d36cf21df74 /var/run/docker/libcontainerd/21d2b053d10
root      1961  1944  0 10:47 ?        00:00:00          \_ /dev/init -s -- sleep 7777
root      1967  1961  0 10:47 ?        00:00:00              \_ sleep 7777

+1

Any updates? We still want this.

mmm, ok, so I agree - I too would expect that docker exec would trap the kill signal and pass it on to the Docker daemon, which should then pass the signal on to the exec’d child

I don’t see much in the way of support for this in the API, http://docs.docker.com/reference/api/docker_remote_api_v1.15/#exec-create, so bug?

@awesomebytes

We use -it and thus your solution did not work for us. (but thanks for the great work).

I wrote my own solution: https://github.com/hackerschoice/segfault/blob/main/host/docker-exec-sigproxy.c

The tool intercepts traffic on the /var/run/docker.sock and detects when a ‘docker exec’ happens. It registers all signals and then forwards (proxies) the signal to the process running inside the instance.

Old command:

docker exec -it alpine

new command:

docker-exec-sigproxy exec -it alpine

I wonder why docker wont add the --sig-proxy=true to the ‘docker exec’…Half the Internet is crying about stale processes and are being suggested to use --init and send down the wrong path…

We’re hitting this as well, both on docker 1.12.6 and on 17.0.6.0-ce. We have some automated processes that attach to running containers via docker exec -it from an SSH session, interacting with the running processes and gathering output to logs. If the SSH connection is disrupted, the docker exec’d process remains running. We end up accumulating these stale processes over time.

@SvenDowideit ah, my use case is that the docker exec process is killed from outside of the container, not the process started by docker exec inside the container. For example, after running docker exec, the tree will look like ( pseudo pids here to illustrate the point ) :

1024 --- docker run -d -it --name test-exec busybox top
1025 --- docker exec -it --name test-exec sh
10 --- docker -d
  \ 10000 --- top
  \ 10001 --- nsenter-exec --nspid 23119 --console /dev/pts/19 -- sh
          \---- sh

Now if i do kill 1025, which kill the docker exec process, the process tree becomes:

1024 --- docker run -d -it --name test-exec busybox top
10 --- docker -d
  \ 10000 --- top
  \ 10001 --- nsenter-exec --nspid 23119 --console /dev/pts/19 -- sh
          \---- sh

I would expect nsenter-exec to be killed as well and/or maybe docker should expose a way to programatically stopped the exec-process from outside.

The new docker exec doc is there https://docs.docker.com/engine/api/v1.43/#tag/Exec/operation/ExecStart

There is still no way to kill a started exec

I ran into the same problem (with hundreds of stale shells not getting SIGHUP when the docker exec client received the SIGHUP). Also, looks like half of the Internet thinks that --init would solve it - obviously not and sends people into the wrong direction.

I wrote it down here: https://gist.github.com/SkyperTHC/cb4ebb633890ac36ad86e80c6c7a9bb2

The workaround at the moment is a clean-up cron job - it’s a mess.

Spent several hours running around in circles yesterday, thanks to this “feature”. (Well with kubectl, but I blame docker for starting this mess).

If it’s not going to be fixed, can we at least have a clear warning in the documentation / --help output / man page?

@th3mis ok, I meant running your command via docker run --init --rm -it --name ubuntu ubuntu:16.04 bash -c 'sleep 77' instead.

If you still want to use docker-exec you have to run your command via the init-process manually: docker exec ubuntu bash -c '/dev/init -s -- sleep 77'

Otherwise the signals are still not forward to your bash process

docker run has a flag --sig-proxy, which is enabled by default when attached.

Not sure about changing defaults, but it would be nice to have docker exec be able to proxy signals to the process it’s attached to.

Good point. We should expose exec jobs belonging to a container.

On Wed, Nov 12, 2014 at 9:59 AM, Sebastiaan van Stijn < notifications@github.com> wrote:

Maybe we can have some list api for exec?

Perhaps add a way to see all processes related to a container? Eg

docker containers ps <containerid>

Which will include the exec process.

— Reply to this email directly or view it on GitHub https://github.com/docker/docker/issues/9098#issuecomment-62762031.