moby: Restoring containers from a custom checkpoint-dir is broken

Hi @harishanand95

The workaround would be to manually copy the checkpoint directory inside /var/lib/docker/containers/<CONTAINER ID>/checkpoints/

For example:

docker run -d --name looper2 busybox \
         /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

docker checkpoint create --checkpoint-dir=/tmp looper2 checkpoint2

mv /tmp/checkpoint2 /var/lib/docker/containers/$(docker ps -aq --no-trunc --filter name=looper2)/checkpoints/

docker start --checkpoint=checkpoint2 looper2

I have a patch to fix this issue. I’m going to create a pull request, when https://github.com/containerd/containerd/pull/2425 will be merged.

@cnnrznn I was able to get this working in a newly created container by copying the checkpoint into the newly created container before running it. The following code snippet worked for me:

docker create --name looper3 busybox
sudo cp -r /tmp/checkpoint2 /var/lib/docker/containers/$(docker ps -aq --no-trunc --filter name=looper3)/checkpoints/
docker start --checkpoint=checkpoint2 looper3

Anyone still using the workaround? Checkpoint does not seem to work if applied to a newly restored container. I tried the workaround posted by @rst0git and the scripts by @MihaelBercic of manually copying the checkpoint.

Steps to reproduce:

docker run --security-opt=seccomp:unconfined --name cr -d busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
docker logs -n1 -f cr # Logs work, shows an increasing number (Ctrl+c to stop)
docker checkpoint create --checkpoint-dir /tmp/ cr checkpoint1
docker stop -t0 cr && docker rm cr
docker create --security-opt=seccomp:unconfined --name cr busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
cp -r /tmp/checkpoint1 /var/lib/docker/containers/$(docker ps -aq --no-trunc --filter name=cr)/checkpoints/
docker start --checkpoint checkpoint1 cr # remove the container to restore in a new one
docker logs -n1 -f cr # Logs work, shows an increasing number (Ctrl+c to stop)
docker checkpoint create --checkpoint-dir /tmp/ cr checkpoint2
Error response from daemon: Cannot checkpoint container cr: OCI runtime pause failed: freezer not supported: openat2 /sys/fs/cgroup/system.slice/docker-d4cae4eaeea3a2d93155029a112fbe4062b7c0402355f662f3ea5ae329441b46.scope/cgroup.freeze: no such file or directory: unknown

The container fails to pause after it is restored in a new container.

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.0.0+unknown)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 20.10.25
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
 runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 6.1.38-59.109.amzn2023.x86_64
 Operating System: Amazon Linux 2023
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 949.8MiB
 Name: ip-10-0-139-156.ec2.internal
 ID: 54UQ:MNHU:YLFT:E5ZG:OJTX:XVYP:YUFB:R2LQ:3HY4:ZERB:OSUU:THDA
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Tested with criu 3.17 and 3.18

criu --version
Version: 3.18

containerd

containerd --version
containerd github.com/containerd/containerd 1.7.2 0cae528dd6cb557f7201036e9f43420650207b58

Hi @MihaelBercic, you should be able to use docker create to first create the container then move the checkpoint.

However, I would recommend using Podman for container migration as it is actively developed, supported and maintained.

@adrianreber is the author of the checkpoint/restore functionality in Podman and also a CRIU maintainer. Adrian has a few very good talks and articles on this topic:

• Container live migration with Podman and CRIU: https://www.youtube.com/watch?v=FwbZuRMdO94
• Container Migration and CRIU Details https://www.youtube.com/watch?v=-7DgNxyuz_o
• Container Live Migration: https://www.youtube.com/watch?v=x7ypRhplRvc
• Container Live Migration https://archive.fosdem.org/2020/schedule/event/containers_live_migration/
• Container migration with Podman: https://www.redhat.com/en/blog/container-migration-podman-rhel

I hope this helps.

@avagin I wonder if that PR is ready to be made now that containerd/containerd#2425 is ready to roll? I can try to tackle it if not - thank you!