moby: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""

Description

Can’t start containers, all of them end up with status created and error indicated in the header

Steps to reproduce the issue: docker run -d --rm hello-world

Describe the results you received:

# docker run -d --rm hello-world
ae74077f652d856037d5a51268529d7d11247a811067340abd671fa71945d755
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\"".

Describe the results you expected: Id of started container, no errors

Additional information you deem important (e.g. issue happens only occasionally): Don’t know what happened. We are running nomad cluster with three clients, one of them couldn’t schedule containers at some point and then we figured out that we can’t run any containers even manually. Tried to upgrade docker to the latest version, Docker got upgraded, but issue persisted.

Just found that getting even more errors when running with less arguments

# docker run hello-world
nsenter: failed to unshare namespaces: Cannot allocate memory
container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\""
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:245: running exec setns process for init caused \"exit status 29\"".

Output of docker version:

]# docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 10
 Running: 0
 Paused: 0
 Stopped: 10
Images: 2
Server Version: 1.13.1
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.632 GiB
Name: server.linux.local
ID: HAQQ:4HVX:HDMO:WB63:DK2O:NRP4:TJTR:2JGK:G5R5:OVUN:A5RL:JHPF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 16
 Goroutines: 22
 System Time: 2017-02-15T10:53:11.563182654+01:00
 EventsListeners: 0
Http Proxy: http://proxy:8080
Https Proxy: http://proxy:8080
No Proxy: localhost,127.0.0.1,registry.service.consul
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 registry.service.consul:5000
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Journal logs

#jounralctl -u docker
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.102068089+01:00" level=debug msg="Calling POST /v1.26/containers/create"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.103376362+01:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":false,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"hello-world\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.163661326+01:00" level=debug msg="container mounted via layerStore: /var/lib/docker/overlay/fc723c65995887f6f61fb9b0d658be868f7dc532ec82f8a30d140a4c07418f0a/merged"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.169605209+01:00" level=debug msg="Calling POST /v1.26/containers/39f9a449fb17a82d37f1872d5df3b302818c93937cc3b374c9ea6da1a3b6c53b/attach?stderr=1&stdout=1&stream=1"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.171057850+01:00" level=debug msg="Calling GET /v1.26/events?filters=%7B%22container%22%3A%7B%2239f9a449fb17a82d37f1872d5df3b302818c93937cc3b374c9ea6da1a3b6c53b%22%3Atrue%7D%2C%22type%22%3A%7B%22container%22%3Atrue%7D%7D"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.171810891+01:00" level=debug msg="attach: stdout: begin"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.171869535+01:00" level=debug msg="attach: stderr: begin"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.172139205+01:00" level=debug msg="Calling POST /v1.26/containers/39f9a449fb17a82d37f1872d5df3b302818c93937cc3b374c9ea6da1a3b6c53b/start"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.174094880+01:00" level=debug msg="container mounted via layerStore: /var/lib/docker/overlay/fc723c65995887f6f61fb9b0d658be868f7dc532ec82f8a30d140a4c07418f0a/merged"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.174522872+01:00" level=debug msg="Assigning addresses for endpoint nervous_mestorf's interface on network bridge"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.174556168+01:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.179626608+01:00" level=debug msg="Assigning addresses for endpoint nervous_mestorf's interface on network bridge"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.185714219+01:00" level=debug msg="Programming external connectivity on endpoint nervous_mestorf (f2161d5e4d418e416d301dafe0cce88a03d362a87d68b581b394252f6f4db872)"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.277852594+01:00" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:245: running exec setns process for init caused \\\"exit status 29\\\"\"\n" id=39f9a449fb17a82d37f1872d5df3b302818c93937cc3b374c9ea6da1a3b6c53b
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.278694094+01:00" level=error msg="Create container failed with error: oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:245: running exec setns process for init caused \\\"exit status 29\\\"\"\n"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.290740930+01:00" level=debug msg="attach: stdout: end"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.290769551+01:00" level=debug msg="attach: stderr: end"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.295119581+01:00" level=debug msg="Revoking external connectivity on endpoint nervous_mestorf (f2161d5e4d418e416d301dafe0cce88a03d362a87d68b581b394252f6f4db872)"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.312889938+01:00" level=debug msg="Releasing addresses for endpoint nervous_mestorf's interface on network bridge"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.312928574+01:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.318038828+01:00" level=error msg="Handler for POST /v1.26/containers/39f9a449fb17a82d37f1872d5df3b302818c93937cc3b374c9ea6da1a3b6c53b/start returned error: oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:245: running exec setns process for init caused \\\"exit status 29\\\"\"\n"
Feb 15 11:35:35 padm183.linux.itv.local dockerd[30692]: time="2017-02-15T11:35:35.326998627+01:00" level=debug msg="Client context cancelled, stop sending events"

# cat /etc/docker/daemon.json
{
        "debug": true,
        "selinux-enabled": false,
        "storage-driver": "overlay",
        "dns": ["server"],
        "ip": "server",
        "iptables": true,
        "insecure-registries": ["registry.service.consul:5000"],
        "hosts": [
                "tcp://server:2375",
                "unix:///var/run/docker.sock"
        ]
}