moby: Can't restore containers

Description

Restoring a container no longer works in the latest version of moby/docker.

Steps to reproduce the issue:

*** Before this works, you may need to upgrade CRIU to a newer version than the one in the standard shell Dockerfile ***

export CRIU_VERSION=3.6; apt-get install libnet-dev -y && \
	mkdir -p /usr/src/criu \
	&& curl -sSL https://github.com/xemul/criu/archive/v${CRIU_VERSION}.tar.gz | tar -v -C /usr/src/criu/ -xz --strip-components=1 \
	&& cd /usr/src/criu \
	&& make \
	&& make install-criu

Create a container: docker run --name crtest -d --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

Create a checkpoint of that container: docker checkpoint create crtest checkpoint1

Stop the container (see #35690): docker stop crtest

Attempt to restore from the checkpoint: docker start --checkpoint checkpoint1 crtest

Additional information you deem important (e.g. issue happens only occasionally):

The error I see output in the logs is this:

ERRO[2017-12-04T18:00:14.593051694Z] 6f75781052e4b95e9f829345bff7bb42365a33f2d06578b84be288a89efa316c cleanup: failed to delete container from containerd: no such container 
ERRO[2017-12-04T18:00:14.593360378Z] Handler for POST /v1.30/containers/6f/start returned error: failed to retrieve OCI runtime container pid: open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/6f75781052e4b95e9f829345bff7bb42365a33f2d06578b84be288a89efa316c/init.pid: no such file or directory: unknown 
Error response from daemon: failed to retrieve OCI runtime container pid: open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/6f75781052e4b95e9f829345bff7bb42365a33f2d06578b84be288a89efa316c/init.pid: no such file or directory: unknown

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:15:15 2017
 OS/Arch:      linux/amd64

Server:
 Version:      dev
 API version:  1.35 (minimum version 1.12)
 Go version:   go1.9.2
 Git commit:   7ef9266
 Built:        Mon Dec  4 17:07:37 2017
 OS/Arch:      linux/amd64
 Experimental: true
Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: dev
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 3
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6bff39c643886dfa3d546e83a90a527b64ddeacf
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Debian GNU/Linux 9 (stretch) (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.855GiB
Name: ddaf0d20627d
ID: NRZK:4RHA:M4ZF:TAVO:ROAJ:AZQZ:YDMD:ZW4J:TEQD:RTBN:XK6F:RC65
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 34
 System Time: 2017-12-04T17:32:49.110767545Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Docker for Mac
Version 17.09.0-ce-mac35 (19611)
Channel: stable
a98b7c1b7c

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 36 (16 by maintainers)

Most upvoted comments

this recently started happening to me on 18.09.8 on ubuntu server 16.04 LTS installations,

Could you describe what is “this”? What version of CRIU do you use? Could you attache criu logs?

I’m having an issue doing c/r with some docker containers. I want to work on custom checkpoint dirs but I can’t find where Docker/Moby/runc/containerd/God stores the CRIU logs. Any pointers?

Thanks 😃

Probably off-topic here, but…to my knowledge, creating checkpoints to custom dirs is fine, but restoring from them is broken (see moby/moby#37344). I work around it by just wrapping restores in a bash script that preps a custom-dir restore by copying the relevant checkpoint to the default location (in my case, /var/lib/docker/containers/ID/checkpoints/) and then deleting it after the container has restored. see https://github.com/moby/moby/issues/37344#issuecomment-450782189

If you mean you want to work on the issue, maybe the other bug report would give some guidance.

@crashtua: I have installed the nightly built and latest version of containerd.io on ubuntu 18.04. However I still get the same issue of missing config.json. /var/lib/docker/containers/b008cef367b8b1e8b6046273b9e33dc5ed0227fb9cbae8ea96945ba138284ca2/checkpoints/20181201_1226/config.json

I am trying this on a fresh install of docker with no legacy, all new containers. Do you have any suggestions as to why it may be failing now?

@tswift242 As far as I know, it isn’t there yet. I asked @kolyshkin to help with updating runc in moby.