runc: Got starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 40\"": unknown. from time to time
os: centos 7.2
uname -a: Linux iZ2ze43t8c42mqytqholpuZ 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
docker -v Docker version 17.12.0-ce, build c97c6d6
docker-runc -v runc version 1.0.0-rc4+dev commit: b2567b37d7b75eb4cf325b77297b140ea686ce8f spec: 1.0.0
When I run docker run or docker build, the following error appears from time to time. The Probability is around 5% .
docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 40\"": unknown.
Any clues?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 26 (5 by maintainers)
It turns out not a problem of docker. The machine was doing a lot of file reading work. So most of the memories are consumed by page cache, which I can tell from the
free -mcommand.After I periodically clean the page cache by running
echo 1 > /proc/sys/vm/drop_caches, the problem disappears.Most of this errors that I found are related to bad mounting of volumes.
in my case I was mounting a file to a folder
- ./kibana/config.yml:/usr/share/kibana/config/having the same issue as given below:
:/home/ubuntu# docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi Unable to find image ‘nvidia/cuda:9.0-base’ locally 9.0-base: Pulling from nvidia/cuda 9ff7e2e5f967: Pull complete 59856638ac9f: Pull complete 6f317d6d954b: Pull complete a9dde5e2a643: Pull complete 3dab314fc51e: Pull complete 1a4e7e8b3753: Pull complete 388ed6e4a282: Pull complete Digest: sha256:09ee586c314e599f7b82317fccfbf4717e037e5b83a9c9a9d7a5ccfe810a3071 Status: Downloaded newer image for nvidia/cuda:9.0-base docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused "process_linux.go:413: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=9.0 --pid=13505 /var/lib/docker/overlay2/88efd338b570e5d09dd7f18fd0b508f962b0060bcdbb448a538b43f9b0b50b66/merged]\\nnvidia-container-cli: initialization error: cuda error: invalid device ordinal\\n\""”: unknown.