moby: Can't restore containers
Description
Restoring a container no longer works in the latest version of moby/docker.
Steps to reproduce the issue:
*** Before this works, you may need to upgrade CRIU to a newer version than the one in the standard shell Dockerfile ***
export CRIU_VERSION=3.6; apt-get install libnet-dev -y && \
mkdir -p /usr/src/criu \
&& curl -sSL https://github.com/xemul/criu/archive/v${CRIU_VERSION}.tar.gz | tar -v -C /usr/src/criu/ -xz --strip-components=1 \
&& cd /usr/src/criu \
&& make \
&& make install-criu
Create a container:
docker run --name crtest -d --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
Create a checkpoint of that container:
docker checkpoint create crtest checkpoint1
Stop the container (see #35690):
docker stop crtest
Attempt to restore from the checkpoint:
docker start --checkpoint checkpoint1 crtest
Additional information you deem important (e.g. issue happens only occasionally):
The error I see output in the logs is this:
ERRO[2017-12-04T18:00:14.593051694Z] 6f75781052e4b95e9f829345bff7bb42365a33f2d06578b84be288a89efa316c cleanup: failed to delete container from containerd: no such container
ERRO[2017-12-04T18:00:14.593360378Z] Handler for POST /v1.30/containers/6f/start returned error: failed to retrieve OCI runtime container pid: open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/6f75781052e4b95e9f829345bff7bb42365a33f2d06578b84be288a89efa316c/init.pid: no such file or directory: unknown
Error response from daemon: failed to retrieve OCI runtime container pid: open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/6f75781052e4b95e9f829345bff7bb42365a33f2d06578b84be288a89efa316c/init.pid: no such file or directory: unknown
Output of docker version:
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:15:15 2017
OS/Arch: linux/amd64
Server:
Version: dev
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: 7ef9266
Built: Mon Dec 4 17:07:37 2017
OS/Arch: linux/amd64
Experimental: true
Output of docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: dev
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 3
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6bff39c643886dfa3d546e83a90a527b64ddeacf
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.49-moby
Operating System: Debian GNU/Linux 9 (stretch) (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.855GiB
Name: ddaf0d20627d
ID: NRZK:4RHA:M4ZF:TAVO:ROAJ:AZQZ:YDMD:ZW4J:TEQD:RTBN:XK6F:RC65
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 19
Goroutines: 34
System Time: 2017-12-04T17:32:49.110767545Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Docker for Mac
Version 17.09.0-ce-mac35 (19611)
Channel: stable
a98b7c1b7c
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 36 (16 by maintainers)
I’m working on this issue:  https://github.com/containerd/containerd/pull/2425
Probably off-topic here, but…to my knowledge, creating checkpoints to custom dirs is fine, but restoring from them is broken (see moby/moby#37344). I work around it by just wrapping restores in a bash script that preps a custom-dir restore by copying the relevant checkpoint to the default location (in my case,
/var/lib/docker/containers/ID/checkpoints/
) and then deleting it after the container has restored. see https://github.com/moby/moby/issues/37344#issuecomment-450782189If you mean you want to work on the issue, maybe the other bug report would give some guidance.
@crashtua: I have installed the nightly built and latest version of containerd.io on ubuntu 18.04. However I still get the same issue of missing config.json. /var/lib/docker/containers/b008cef367b8b1e8b6046273b9e33dc5ed0227fb9cbae8ea96945ba138284ca2/checkpoints/20181201_1226/config.json
I am trying this on a fresh install of docker with no legacy, all new containers. Do you have any suggestions as to why it may be failing now?
@tswift242 As far as I know, it isn’t there yet. I asked @kolyshkin to help with updating runc in moby.
This issue was fixed in runc: https://github.com/opencontainers/runc/commit/8187fb740c202f5d29f1717bb933143c10dae8a1