cri-o: crio jams up kubelet on restart if container takes a long time to terminate
What happened?
When a container takes a long time to terminate, and kubelet is restarted, kubelet hangs until the container terminates. This makes the whole node unready. I tracked this down to when cadvisor is started, and then to somewhere in newCrioContainerHandler, probably when it gets info about the container
In our setup, we have a certain workload that consists of a pod with two containers, and one of those eats sigterm and just waits for a long time to shut down. That container is the one that causes this whole mess.
Kubelet will just keep logging:
kubelet.service[2043933]: E1108 18:54:46.297244 2043933 kubelet.go:2383] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
kubelet.service[2043933]: I1108 18:54:46.320642 2043933 kuberuntime_manager.go:447] "Retrieved pods from runtime" all=true
As for logs, here’s the last log that makes it in before the long wait for the container to shut down:
Nov 07 19:53:23 ip-172-16-111-198 kubelet.service[1162146]: I1107 19:53:23.354625 1162146 factory.go:275] Using factory "crio" for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d269700_ddd7_4c51_ba9e_40c014193013.slice/crio-0b74fdbd9af1713bb6abf1bfce22715675e13dd70b3e9c0fa2c312b63400186d"
and after the container shuts down, we see:
Nov 07 19:57:16 ip-172-16-111-198 kubelet.service[1162146]: I1107 19:57:16.559026 1162146 manager.go:981] Added container: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d269700_ddd7_4c51_ba9e_40c014193013.slice/crio-0b74fdbd9af1713bb6abf1bfce22715675e13dd70b3e9c0fa2c312b63400186d" (aliases: [k8s_POD_session-agent-6fdcdc7968-rvp7q_browsers-lighthouse_0d269700-ddd7-4c51-ba9e-40c014193013_0 0b74fdbd9af1713bb6abf1bfce22715675e13dd70b3e9c0fa2c312b63400186d], namespace: "crio")
which supports the theory that it’s cadvisor hanging on getting the container info from crio.
Interestingly, crictl inspect works just fine
I got a goroutine dump from crio while it’s hanging after the kubelet restart: here
What did you expect to happen?
Crio should not hang and kubelet should be able to start up
How can we reproduce it (as minimally and precisely as possible)?
Create a pod with two containers, one of which eats sigterm:
apiVersion: v1
kind: Pod
metadata:
annotations:
generateName: test-pod-
spec:
terminationGracePeriodSeconds: 86300
containers:
- name: fast-shutdown
command: ["bash", "-c", "sleep infinity"]
- name: never-shutdown
command: ["bash", "-c", "trap 'sleep 100000' SIGTERM; sleep infinity"]
Delete the pod, and then restart kubelet on the node
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
crio version 1.29.4
Version: 1.29.4
GitCommit: 51ea93e0b9af5ad2cfa7f8071ec48d99bf39a3ec
GitCommitDate: 2024-04-30T06:05:39Z
GitTreeState: clean
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.21.7
Compiler: gc
Platform: linux/amd64
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
exclude_graphdriver_devicemapper
seccomp
apparmor
selinux
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false
$ kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "29",
"gitVersion": "v1.29.10",
"gitCommit": "f0c1ea863533246b6d3fda3e6addb7c13c8a6359",
"gitTreeState": "archive",
"buildDate": "2024-11-04T18:09:18Z",
"goVersion": "go1.22.8",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3"
}
OS version
# On Linux:
$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
$ uname -a
Linux ip-172-18-59-83 6.1.109-hs83.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 24 17:33:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Additional environment details (AWS, VirtualBox, physical, etc.)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 29 (17 by maintainers)
We should only need it in 1.30 AFAIR, and that backport is in progress
I found that this PR: https://github.com/google/cadvisor/pull/3457 fixed my problem!
With cadvisor a https://github.com/google/cadvisor/commit/42bb3d13a0cf9ab80c880a16c4ebb4f36e51b0c9 - it works, and with the previous commit https://github.com/google/cadvisor/commit/27f1e92b0bcc9d97c10fdc5e6b808e2847e907fb it does not