cri-o: crio is crashed during container restore

What happened?

  1. I’m following this procedure to test container checkpoint/restore with Kubernetes 1.30: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone
  2. After I created checkpoint of container started by Kubernetes successfully, I deleted Kubernetes cluster, and try to restore a container via crio.
  3. I used following json to create pod sandbox: crictl runp pod-config.json
{
    "metadata": {
        "name": "clone-sandbox",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    "linux": {
    }
}
  1. Then I used following json to restore container from checkpoint: crictl create ef80f5a8c32b7 container-config.json pod-config.json
{
  "metadata": {
      "name": "test"
  },
  "image":{
      "image": "/home/tonyaw/k8s/kubelet/checkpoints/checkpoint-tonyaw-deploy-78fdd8bbf5-5p2zf_default-test-2024-04-29T20:43:15+08:00.tar"
  }
}
  1. crictl is failed, and crio is crashed:
[root@test-host kubelet]# crictl create 96480f6885a14 container-config.json pod-config.json
 E0429 20:50:58.512622   44825 remote_runtime.go:319] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Dead
 lineExceeded desc = context deadline exceeded" podSandboxID="96480f6885a14"

crio journal log and backtrace: crio-crash-log.txt

What did you expect to happen?

container is restored successfully.

How can we reproduce it (as minimally and precisely as possible)?

Follow procedure https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone to restore a container with Kubernetes 1.30. The issue shall happen when you restore the container.

Anything else we need to know?

No response

CRI-O and Kubernetes version

$ crio --version
WARN[0000] Failed to decode the keys ["storage.options.override_kernel_check"] from "/etc/containers/storage.conf"
crio version 1.30.0
Version:        1.30.0
GitCommit:      a3de11a8aa3070c8b040054461ce82314c95baf2
GitCommitDate:  2024-04-26T16:51:03Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.22.0
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  false
$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "30",
    "gitVersion": "v1.30.0",
    "gitCommit": "7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a",
    "gitTreeState": "clean",
    "buildDate": "2024-04-17T17:36:05Z",
    "goVersion": "go1.22.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3"
}
The connection to the server 10.67.39.251:6443 was refused - did you specify the right host or port?

I deleted Kubernetes cluster, as crictl can’t create pod successfully if Kubernetes cluster is up with calico.

OS version

# On Linux:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
$ uname -a
Linux foss-ssc-11 5.1.11-1.el7.elrepo.x86_64 #1 SMP Mon Jun 17 15:51:08 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 37 (22 by maintainers)

Most upvoted comments

Thanks for your detail info! I will leave the ticket open to fix of original issue, and you can close the ticket if you want. 😃

  1. pod can be restored successfully:

Great.

Looks like “pid 1 can’t be a shell” is the root case. 😄

This is technically not correct. In your current example your PID 1is also a shell:

1 root       0:00 sh -c while true; do echo "$(date)" >> test.log; sleep 1; done

So far no one was using stdin: true. Usually stdin is connected to /dev/null in your case it was connected to a pipe.

If there are multiple process running in the container, will all the processes will be checkpoint/restore, or only pid 1? 😃

Yes, if those processes are child processes of PID 1. CRIU always checkpoints a complete process tree. The target PID and all child processes.

It is important to know that crictl exec or kubectl exec creates processes which are not child processes of PID 1. So if you exec into a container and start additional processes they will not be checkpointed because they are not child processes of PID 1.

mnt: 		Will mount 2233 from /var/lib/kubelet/pods/51e3eb40-752e-4eeb-91cb-4bb61f90868b/volumes/kubernetes.io~projected/kube-api-access-b78wn (E)
mnt: 		Will mount 2233 @ /tmp/.criu.mntns.PmL0Et/13-0000000000/run/secrets/kubernetes.io/serviceaccount
mnt: 	Read 2233 mp @ /tmp/.criu.mntns.PmL0Et/13-0000000000/run/secrets/kubernetes.io/serviceaccount
Error (criu/mount.c:2830): mnt: No mapping for /run/secrets mountpoint

@tonyaw The error that appears during crictl start occurs because the mountpoint /run/secrets has not been created. CRI-O needs to be configured to create this mountpoint.

So actually, it is impossible. Right?

I believe Adrian recommends restoring with Kubernetes because it would be easier. While restoring a container outside of Kubernetes is possible, it requires additional configurations that Kubernetes would otherwise handle automatically.

Now the restore of CRIU failed. Now you need to provide us with: /run/containers/storage/overlay-containers/cdc9864e3a5c73437de70fb52343a7798a5bd0b3a86b429d7400e26c02c83e55/userdata/restore.log.

This file is usually deleted by CRI-O. You need to tell CRIU to create the log file in a different location.

# echo "log-file /tmp/criu.log" >> /etc/criu/runc.conf

And then share /tmp/criu.log with us.

1. What CentOS verison do you suggest, is CentOS8 good enough?

Yes, that should work.

2. Is there kernel version required?

The CentOS 8 kernel has been tested to work as far as I know.

3. For the "pod-config.json" and "container-config.json" I provided, any change can be made to make it work even with crio 1.30?

For testing (see https://github.com/cri-o/cri-o/blob/main/test/checkpoint.bats#L45) we use the following two files:

Looking at @kwilczynski comment you need the linux part of container_sleep.json to make it work. linux: {} could be all there is needed. I think https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone does not provide the complete JSON file, just the important parts that you need to change.

@rst0git restoring from a tar file is supported, just not from Kubernetes. Using crictl that is possible.

@kwilczynski You are right, there should be an additional check. Thanks. I can also open a PR if you want.