moby: Random "Cannot start container" Errors on 1.9.0-rc5 CentOS7

Since upgrading to 1.9.0 RCs on a CentOS7 host (3.10.0-229.14.1.el7.x86_64), I’ve been seeing random Cannot start container errors, e.g.:

Error response from daemon: Cannot start container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: [8] System error: write /sys/fs/cgroup/devices/system.slice/docker-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.scope/cgroup.procs: no such device

Relevant log entries are below for reference:

 Nov  3 08:48:10 dockerhost systemd: Starting docker container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.
Nov  3 08:48:10 dockerhost systemd: Started docker container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.510245551-05:00" level=warning msg="signal: killed"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.551716997-05:00" level=debug msg="Releasing addresses for endpoint adoring_bhabha's interface on network bridge"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.551781175-05:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.4)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.553129616-05:00" level=debug msg="[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.553166270-05:00" level=debug msg="[devmapper] Unmount(/data/docker/devicemapper/mnt/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.555363308-05:00" level=debug msg="[devmapper] Unmount done"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.555390143-05:00" level=debug msg="[devmapper] deactivateDevice(479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.555473755-05:00" level=debug msg="[devmapper] removeDevice START(docker-253:4-1074522036-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost systemd: Unit iscsi.service cannot be reloaded because it is inactive.
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.593476956-05:00" level=debug msg="[devmapper] removeDevice END(docker-253:4-1074522036-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.593519481-05:00" level=debug msg="[devmapper] deactivateDevice END(479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.593543175-05:00" level=debug msg="[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b) END"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.594937237-05:00" level=error msg="failed to umount /data/docker/containers/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b/shm: invalid argument"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.594989602-05:00" level=error msg="failed to umount /data/docker/containers/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b/mqueue: invalid argument"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.595014866-05:00" level=error msg="479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: Failed to umount ipc filesystems: failed to cleanup ipc mounts:\ninvalid argument\ninvalid argument"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.595046108-05:00" level=debug msg="[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.595067916-05:00" level=debug msg="[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b) END"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.595093237-05:00" level=error msg="Error unmounting device 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: UnmountDevice: device not-mounted id 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.595199876-05:00" level=error msg="Handler for POST /v1.21/containers/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b/start returned error: Cannot start container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: [8] System error: write /sys/fs/cgroup/devices/system.slice/docker-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.scope/cgroup.procs: no such device"
Nov  3 08:48:10 dockerhost docker: time="2015-11-03T08:48:10.595236064-05:00" level=error msg="HTTP Error" err="Cannot start container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: [8] System error: write /sys/fs/cgroup/devices/system.slice/docker-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.scope/cgroup.procs: no such device" statusCode=500

I’m able to reproduce this fairly reliably on CentOS7 using the technique mentioned in #17387:

 (set -e ; while true; do ID=$(docker run -d ibuildthecloud/helloworld:latest); docker rm -fv $ID;done)

This issue does not occur with Docker 1.8.3 installed, nor can I reproduce it with 1.9.0-rc5 on a system without systemd using an AUFS storage backend (Ubuntu 14.04).

Bug Report Information

$ docker version
Client:
 Version:      1.9.0-rc5
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   9318004
 Built:        Tue Nov  3 06:14:13 UTC 2015
 OS/Arch:      linux/amd64
Server:
 Version:      1.9.0-rc5
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   9318004
 Built:        Tue Nov  3 06:14:13 UTC 2015
 OS/Arch:      linux/amd64
$ docker info
Containers: 47
Images: 731
Server Version: 1.9.0-rc5
Storage Driver: devicemapper
Pool Name: docker-253:4-1074522036-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: xfs
Data file: /dev/centos_docker/docker_data
Metadata file: /dev/centos_docker/docker_metadata
Data Space Used: 88.79 GB
Data Space Total: 614.9 GB
Data Space Available: 526.1 GB
Metadata Space Used: 120 MB
Metadata Space Total: 17.05 GB
Metadata Space Available: 16.93 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 32
Total Memory: 188.7 GiB
Name: dockerhost
ID: IBQH:7AOM:H6KR:XXK5:5JKT:GODZ:EUEM:7VF2:6THK:W7NQ:SOPU:DL5Y
Debug mode (server): true
   File Descriptors: 33
   Goroutines: 59
   System Time: 2015-11-03T11:38:44.424354397-05:00
   EventsListeners: 0
   Init SHA1: d676303f7af5bf0ded806f3e60d02b299cd01060
   Init Path: /usr/libexec/docker/dockerinit
   Docker Root Dir: /data/docker
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
$ uname -a
 Linux dockerhost 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 41 (14 by maintainers)

Commits related to this issue

Workaround for Random Cannot start container https://github.com/docker/docker/issues/17653 — committed to hortonworks/cloudbreak-images by akanto 8 years ago
Workaround for Random Cannot start container https://github.com/docker/docker/issues/17653 — committed to hortonworks/cloudbreak-images by akanto 8 years ago

Most upvoted comments

So far I can reproduce but cannot find the root cause.

A workaround for everyone is to launch the daemon with --exec-opt native.cgroupdriver=cgroupfs to not use systemd as the cgroups. Your unit file would look like this.

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon --exec-opt native.cgroupdriver=cgroupfs -H fd:// 
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

Sorry about not finding a better fix, i’ll keep looking for clues. However, if you make this change, i have been running tests for a day now and there are zero errors.

+13

crosbymichael on Nov 11, 2015