cri-o: Containers can get stuck in stopping state until cri-o is restarted

What happened?

A pod is deleted and kubelet tells the container to stop

Mar 02 19:05:32 ip-172-18-52-92 crio[3133611]: time="2023-03-02 19:05:32.885463979Z" level=info msg="Stopping container: 252c0f00d0e3f9ff99016b2dbc7fe4c2877b29d53f6d8a66dbc0fb81710ae8a9 (timeout: 30s)" id=4ee0306c-e4b3-43c1-9f1f-747364e6edb1 name=/runtime.v1.RuntimeService/StopContainer

The stop times out

Mar 02 19:06:03 ip-172-18-52-92 crio[3133611]: time="2023-03-02 19:06:03.124419954Z" level=warning msg="Stopping container 252c0f00d0e3f9ff99016b2dbc7fe4c2877b29d53f6d8a66dbc0fb81710ae8a9 with stop signal timed out: timeout reached after 30 seconds waiting for container process to exit" id=4ee0306c-e4b3-43c1-9f1f-747364e6edb1 name=/runtime.v1.RuntimeService/StopContainer

But it appears that the cgroups, etc get torn down anyway:

Mar 02 19:08:02 ip-172-18-52-92 crio[3133611]: time="2023-03-02 19:08:02.885652165Z" level=warning msg="Failed to retrieve cgroup stats: open /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ae14ac_623d_4685_a19b_2c6ecf03d24f.slice/crio-252c0f00d0e3f9ff99016b2dbc7fe4c2877b29d53f6d8a66dbc0fb81710ae8a9.scope/memory.usage_in_bytes: no such file or directory"
Mar 02 19:08:02 ip-172-18-52-92 crio[3133611]: time="2023-03-02 19:08:02.885701842Z" level=warning msg="Failed to retrieve cgroup stats: open /sys/fs/cgroup/pids/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ae14ac_623d_4685_a19b_2c6ecf03d24f.slice/crio-252c0f00d0e3f9ff99016b2dbc7fe4c2877b29d53f6d8a66dbc0fb81710ae8a9.scope/pids.current: no such file or directory"

In this state, crictl inspect hangs for 30s before showing output (probably due to similar locking issues from https://github.com/cri-o/cri-o/issues/1824):

crictl inspect 252c0f00d0e3f

But once it does show the output, it’s clear that it still agrees that the container is running:

    "state": "CONTAINER_RUNNING",
    "createdAt": "2023-03-02T16:26:56.000421976Z",
    "startedAt": "2023-03-02T16:26:56.64685835Z",
    "finishedAt": "0001-01-01T00:00:00Z",

ps shows that the process is still running:

# ps aux | grep 4037286
root     4037286  0.0  0.0      0     0 ?        Ss   16:26   0:00 [hs-init]

(hs-init there is actually the catatonit binary)

That process is in a weird state though where stuff like stat /proc/4037286/ns/mnt return “not found” errors and issuing kill -9 to the process does not result in it going away. It’s acting like the zombie process state, but the process is actually in the Sleeping state rather than zombie. That process had no children.

Upon issuing systemctl restart crio, crio proceeds to happily remove that container (and the other sidecars for the pod that had already stopped successfully) and the process is no longer present in ps:

Mar 02 20:04:59 ip-172-18-52-92 systemd[1]: Started Container Runtime Interface for OCI (CRI-O).
Mar 02 20:05:11 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:11.244414510Z" level=info msg="Stopping pod sandbox: c826ce942387614e0910805bd1b9e26c7dc11140fc4ed6a5f0a69432e45e6f7f" id=cfdf7818-6a84-4a63-a259-6996f3fc6bc6 name=/runtime.v1.RuntimeService/StopPodSandbox
Mar 02 20:05:11 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:11.246671692Z" level=info msg="Stopped pod sandbox: c826ce942387614e0910805bd1b9e26c7dc11140fc4ed6a5f0a69432e45e6f7f" id=cfdf7818-6a84-4a63-a259-6996f3fc6bc6 name=/runtime.v1.RuntimeService/StopPodSandbox
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.456919058Z" level=info msg="Removing container: 416eb6feb1891101c5bb8a1ed2d7d7b87e618ee9e6da8838955e9517c27ec437" id=6f98bc95-e0cb-45cf-b26d-b0fe76c38a83 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.495261549Z" level=info msg="Removed container 416eb6feb1891101c5bb8a1ed2d7d7b87e618ee9e6da8838955e9517c27ec437: <redacted>" id=6f98bc95-e0cb-45cf-b26d-b0fe76c38a83 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.496281062Z" level=info msg="Removing container: 9345fdae835cbcb78f513b23554582f2d2fc09f0c9078cc8607dc2eb4429f5ca" id=5c0ba04f-e54e-4ac7-965b-eb86e065c0d1 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.570546897Z" level=info msg="Removed container 9345fdae835cbcb78f513b23554582f2d2fc09f0c9078cc8607dc2eb4429f5ca: <redacted>" id=5c0ba04f-e54e-4ac7-965b-eb86e065c0d1 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.571434744Z" level=info msg="Removing container: f335d5ec10ddfa40ae1c95572117289e2a96a54c966ff59d5bb7323b6803e54e" id=3692a894-fbe3-4c86-8ae8-40ceeebbdef3 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.615743420Z" level=info msg="Removed container f335d5ec10ddfa40ae1c95572117289e2a96a54c966ff59d5bb7323b6803e54e: <redacted>" id=3692a894-fbe3-4c86-8ae8-40ceeebbdef3 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.616642517Z" level=info msg="Removing container: 779eaf19273567910b78c3e690f01eab6b408704e0b1f8dd587d799561d611fc" id=fb08d136-93e4-41e7-926a-a409e44175df name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.637593221Z" level=info msg="Removed container 779eaf19273567910b78c3e690f01eab6b408704e0b1f8dd587d799561d611fc: <redacted>" id=fb08d136-93e4-41e7-926a-a409e44175df name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.638671682Z" level=info msg="Removing container: 445a74d8827c66857f039b1a4ab1acab61d83626482eeb9294153a24641ac461" id=a99b07cd-5538-4dcf-a422-933d1ae090d0 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.715996763Z" level=info msg="Removed container 445a74d8827c66857f039b1a4ab1acab61d83626482eeb9294153a24641ac461: <redacted>" id=a99b07cd-5538-4dcf-a422-933d1ae090d0 name=/runtime.v1.RuntimeService/RemoveContainer
Mar 02 20:05:16 ip-172-18-52-92 crio[1364180]: time="2023-03-02 20:05:16.716903728Z" level=info msg="Removing container: 252c0f00d0e3f9ff99016b2dbc7fe4c2877b29d53f6d8a66dbc0fb81710ae8a9" id=6926805f-230a-43ad-8bef-66e9a4398c03 name=/runtime.v1.RuntimeService/RemoveContainer

What did you expect to happen?

Container termination to go through without getting stuck

How can we reproduce it (as minimally and precisely as possible)?

Unfortunately, I do not know. We are in the process of migrating to cri-o and this issue pops up on random pods maybe once a week across ~200 nodes that we have migrated to cri-o.

The one interesting theme that has emerged is that in at least two of the cases, engineers had kubectl exec shells open in the container at the time.

Anything else we need to know?

No response

CRI-O and Kubernetes version

crio version 1.23.4
Version:          1.23.4
GitCommit:        a6a1e6ebf4c4baca7ca7d57f069a4e0cd89056cd
GitTreeState:     dirty
BuildDate:        1980-01-01T00:00:00Z
GoVersion:        go1.17.4
Compiler:         gc
Platform:         linux/amd64
Linkmode:         static
BuildTags:        apparmor, exclude_graphdriver_devicemapper, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  false

(Note that this is the static build of cri-o from the cri-o website)

# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"archive", BuildDate:"2023-03-01T14:28:40Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.16", GitCommit:"60e5135f758b6e43d0523b3277e8d34b4ab3801f", GitTreeState:"archive", BuildDate:"2023-03-01T14:28:40Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

OS version

NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Additional environment details (AWS, VirtualBox, physical, etc.)

# crun --version
crun version 1.5
commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
# conmon --version
conmon version 2.1.3
commit: 691a5fac24834831d2509b3489c416c26e8037c2

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 25 (11 by maintainers)

Most upvoted comments

Definitely not stale, we hit this in downstream OCP with a customer today

thanks for taking a look @terinjokes ! I reopened that other one. I think adding CommandContext is a good first step. Did you want to open a PR to add that?

I took a look at this on Friday and this morning, comparing behavior between containerd and cri-o and I’ve noticed two things:

  1. The lifecycle of the command is disconnected from the lifecycle of the context. The containerd implementation calls CommandContext from os/exec, while cri-o calls Command without separate logic to kill the command if the context is canceled. (There’s also “backup” behavior in containerd to kill the command if an exec timeout expires, if one is configured).
  2. The lifecycle of the context seems disconnected from the SPDY stream. Even if we fix the above, the context passed to ExecContainer doesn’t seem to get canceled when the client disconnects, even when setting stream_idle_timeout to a low value. There was a previous issue about this, which I asked to be reopened.