containerd: shim v2 hangs on reboot/shutdown

Description

Containerd is shipping with KillMode=process in systemd unit, so that shims won’t be killed if containerd stops. Systemd broadcast SIGTERM during the final stage of shutdown, this is the only chance for shim to terminate gracefully. It seems that shim v2 does not handle SIGTERM/SIGINT at all, and hangs machine reboot/shutdown for 90s, got killed at last.

Shim v1 handles SIGINT/SIGTERM https://github.com/containerd/containerd/blob/c7e4747cfb5cf15eef68af71b0a5526f2343f635/cmd/containerd-shim/main_unix.go#L248-L261

Shim v2 registers and ignores SIGINT/SIGTERM https://github.com/containerd/containerd/blob/c7e4747cfb5cf15eef68af71b0a5526f2343f635/runtime/v2/shim/shim_unix.go#L81-L87

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 22
Comments: 16 (7 by maintainers)

Most upvoted comments

To fix this problem I ended up creating a systemd service: kill-all-containers.sh:

#! /usr/bin/env sh
docker ps --format '{{.ID}}' | xargs docker kill

kill-all-containers.service

[Unit]
Before=shutdown.target reboot.target halt.target final.target
DefaultDependencies=false
Description=Kill all docker containers to prevent shutdown lag
RequiresMountFor=/

[Service]
ExecStart=/path/to/kill-all-containers.sh
Type=oneshot
RemainAfterExit=true

[Install]
WantedBy=shutdown.target reboot.target halt.target final.target

Or if you are using nixos, put this in kill-all-docker-containers.nix, and import it in your configuration.nix file

{ config, pkgs, ... }:
{
systemd.services = {
    kill-all-docker-containers = {
        description = "Kill all docker containers to prevent shutdown lag";
        enable = true;
        unitConfig = {
          DefaultDependencies = false;
          RequiresMountFor = "/";
        };
        before = [ "shutdown.target" "reboot.target" "halt.target" "final.target" ];
        wantedBy = [ "shutdown.target" "reboot.target" "halt.target" "final.target" ];
        serviceConfig = {
          Type = "oneshot";
          RemainAfterExit = true;
          ExecStart = pkgs.writeScript "docker-kill-all" ''
            #! ${pkgs.runtimeShell} -e
            ${pkgs.docker}/bin/docker ps --format '{{.ID}}' | xargs ${pkgs.docker}/bin/docker kill
          '';
        };
    };
  };
}

vividn on Jan 24, 2022

I can work on this.

/assign

qiutongs on Jul 30, 2021

Also waiting a long time for the fix in standard debian packages.

AInteriorB on Feb 1, 2022

Still a problem… Need to stop all my containers before a shutdown / restart to avoid the hang

codevski on Jan 21, 2022

Thinking of different things we could do for this… probably it would just be best to shutdown on SIGTERM/SIGINT. I’m not sure if ignoring the signals was intentional.

cpuguy83 on Jun 2, 2021

That will be released as a minor fix change or it must goes public at 1.5?

wiltonlcsj on Jan 31, 2022

PTAL https://github.com/containerd/containerd/pull/5828

It looks like indeed the handling which ignores sigint/sigterm is from the original v2 shim code.

cpuguy83 on Aug 4, 2021

My machine hangs because of this; a fix would be sweet.

JeremyCaron on Jul 19, 2021

Sorry for misunderstanding the issue. I was thinking that the /run folder is not tmpfs so that containerd takes long time to reload the dead shim.

I am fine with the proposal about handling the sigterm in runc-shim-v2.

fuweid on Jun 4, 2021