cri-o: crio is crashed during container restore
What happened?
- I’m following this procedure to test container checkpoint/restore with Kubernetes 1.30: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone
- After I created checkpoint of container started by Kubernetes successfully, I deleted Kubernetes cluster, and try to restore a container via crio.
- I used following json to create pod sandbox: crictl runp pod-config.json
{
"metadata": {
"name": "clone-sandbox",
"namespace": "default",
"attempt": 1,
"uid": "hdishd83djaidwnduwk28bcsb"
},
"linux": {
}
}
- Then I used following json to restore container from checkpoint: crictl create ef80f5a8c32b7 container-config.json pod-config.json
{
"metadata": {
"name": "test"
},
"image":{
"image": "/home/tonyaw/k8s/kubelet/checkpoints/checkpoint-tonyaw-deploy-78fdd8bbf5-5p2zf_default-test-2024-04-29T20:43:15+08:00.tar"
}
}
- crictl is failed, and crio is crashed:
[root@test-host kubelet]# crictl create 96480f6885a14 container-config.json pod-config.json
E0429 20:50:58.512622 44825 remote_runtime.go:319] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Dead
lineExceeded desc = context deadline exceeded" podSandboxID="96480f6885a14"
crio journal log and backtrace: crio-crash-log.txt
What did you expect to happen?
container is restored successfully.
How can we reproduce it (as minimally and precisely as possible)?
Follow procedure https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone to restore a container with Kubernetes 1.30. The issue shall happen when you restore the container.
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
WARN[0000] Failed to decode the keys ["storage.options.override_kernel_check"] from "/etc/containers/storage.conf"
crio version 1.30.0
Version: 1.30.0
GitCommit: a3de11a8aa3070c8b040054461ce82314c95baf2
GitCommitDate: 2024-04-26T16:51:03Z
GitTreeState: clean
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.22.0
Compiler: gc
Platform: linux/amd64
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
exclude_graphdriver_devicemapper
seccomp
apparmor
selinux
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false
$ kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "30",
"gitVersion": "v1.30.0",
"gitCommit": "7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a",
"gitTreeState": "clean",
"buildDate": "2024-04-17T17:36:05Z",
"goVersion": "go1.22.2",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3"
}
The connection to the server 10.67.39.251:6443 was refused - did you specify the right host or port?
I deleted Kubernetes cluster, as crictl can’t create pod successfully if Kubernetes cluster is up with calico.
OS version
# On Linux:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
$ uname -a
Linux foss-ssc-11 5.1.11-1.el7.elrepo.x86_64 #1 SMP Mon Jun 17 15:51:08 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
Additional environment details (AWS, VirtualBox, physical, etc.)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 37 (22 by maintainers)
Thanks for your detail info! I will leave the ticket open to fix of original issue, and you can close the ticket if you want. 😃
Great.
This is technically not correct. In your current example your PID 1is also a shell:
So far no one was using
stdin: true. Usually stdin is connected to/dev/nullin your case it was connected to a pipe.Yes, if those processes are child processes of PID 1. CRIU always checkpoints a complete process tree. The target PID and all child processes.
It is important to know that
crictl execorkubectl execcreates processes which are not child processes of PID 1. So if you exec into a container and start additional processes they will not be checkpointed because they are not child processes of PID 1.@tonyaw The error that appears during
crictl startoccurs because the mountpoint/run/secretshas not been created. CRI-O needs to be configured to create this mountpoint.I believe Adrian recommends restoring with Kubernetes because it would be easier. While restoring a container outside of Kubernetes is possible, it requires additional configurations that Kubernetes would otherwise handle automatically.
Now the restore of CRIU failed. Now you need to provide us with:
/run/containers/storage/overlay-containers/cdc9864e3a5c73437de70fb52343a7798a5bd0b3a86b429d7400e26c02c83e55/userdata/restore.log.This file is usually deleted by CRI-O. You need to tell CRIU to create the log file in a different location.
And then share
/tmp/criu.logwith us.Yes, that should work.
The CentOS 8 kernel has been tested to work as far as I know.
For testing (see https://github.com/cri-o/cri-o/blob/main/test/checkpoint.bats#L45) we use the following two files:
Looking at @kwilczynski comment you need the
linuxpart ofcontainer_sleep.jsonto make it work.linux: {}could be all there is needed. I think https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone does not provide the complete JSON file, just the important parts that you need to change.@rst0git restoring from a tar file is supported, just not from Kubernetes. Using
crictlthat is possible.@kwilczynski You are right, there should be an additional check. Thanks. I can also open a PR if you want.