I have reported this to Ubuntu as a bug in the docker.io package.

Please upvote “This bug affects you” if applicable.

Wow!!

I just read one more time the post by @cpuguy83 and in particular this part:

If you feel like you are running runc v1.0.2 already, please validate:

• That you are using the correct runtime name
• That runc in $PATH (as the containerd daemon process can see it) is what you expect it to be.

The runtime surely is not nvidia and none of containers have it, just double checked. The $PATH surely has runc in it, as runc -v shows the info quoted above however, I decided to run whereis runc and was quite amazed when saw next result:

runc: /usr/bin/runc /opt/containerd/bin/runc

WHAT? I haven’t installed it into /opt. /opt/containerd/bin/runc -v shown me the version is less than 1.0. I just deleted this folder and restarted the laptop (probably just restart of docker service would be enough, but I had another reason to restart the machine as well). Docker and docker-compose works fine now!

@nmtsousa , check if you have similar case, pls.

Same problem with ubuntu 20.04. My solution was to downgrade docker with: sudo apt-get install docker.io=20.10.7-0ubuntu1~20.04.2

I have the same problem with the latest docker.io and even on the latest docker-ce on Ubuntu 18.04 on ESXI. By the way I don´t have any nvidia things installed.

Maybe the following steps will help as long as there is no solution for this problem, I used the old version of docker.io. The official Ubuntu Repo didn´t provide it any longer, so i got it from:

http://repos.reduniv.edu.cu/UBUNTU/ubuntu/pool/universe/d/docker.io/

I installed it with:

sudo dpkg -i /path.to/docker.io_20.10.7-0ubuntu1~18.04.2_amd64.deb

Because of the ubuntu unattended-upgrades it will be changed to the most recent version as soon as possible. But I didn´t change the unattended-upgrades. I just took docker.io out of this.

I did this with:

sudo apt-mark hold docker.io

So it will not be updated till the problem is solved and is unhold with:

sudo apt-mark unhold docker.io

Maybe this helps.

I can confirm that Nvidia has released upgraded packages, which appear to be working fine with docker.io 20.10.7-0ubuntu5~20.04.2. My problem has now been resolved.

# apt-get upgrade -y

docker.io/focal-updates,focal-security 20.10.7-0ubuntu5~20.04.2 amd64 [upgradable from: 20.10.7-0ubuntu1~20.04.2]
libnvidia-container-tools/bionic 1.6.0-1 amd64 [upgradable from: 1.5.1-1]
libnvidia-container1/bionic 1.6.0-1 amd64 [upgradable from: 1.5.1-1]
nvidia-container-runtime/bionic 3.6.0-1 amd64 [upgradable from: 3.5.0-1]
nvidia-container-toolkit/bionic 1.6.0-1 amd64 [upgradable from: 1.5.1-1]
nvidia-docker2/bionic 2.7.0-1 all [upgradable from: 2.6.0-1]

# dockerd --version
Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.2

# nvidia-docker run <args> ; echo $?
eb0d0259ee5f46c448b7d1878aa262c7feba9012e8acbd52cd2bd8ecfb05849f
0

# docker inspect <name> | grep Runtime
            "Runtime": "nvidia",

Are you seeing the same if you install the docker-ce, docker-ce-cli and containerd.io packages from https://download.docker.com ?

(install instructions: https://docs.docker.com/engine/install/ubuntu/)

I followed the instructions on the site, removing all existing versions and installing docker-ce, docker-ce-cli and conainerd.io packages, however the problem persists.

$ docker run --rm -ti alpine docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:348: starting container process caused "error adding seccomp filter rule for syscall clone3: permission denied": unknown.

If I downgrade the docker-ce to the previous version with:

apt-get install docker-ce=5:20.10.9~3-0~ubuntu-focal

Everything works:

$ docker run --rm -ti alpine / #

Same problem with ubuntu 20.04. My solution was to downgrade docker with: sudo apt-get install docker.io=20.10.7-0ubuntu1~20.04.2

This has fixed it for me as well on 20.04.3 LTS. Note that the error was specific to docker --runtime nvidia; using the default runtime works fine.

Before the downgrade:

# dockerd --version
Docker version 20.10.7, build 20.10.7-0ubuntu5~20.04.1

# nvidia-docker run <args> ; echo $?
78cc8d1195126d9e40b087bc36e0608b7fd03a0722c50cb642a4df14699af566
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown.
126

Downgrade

# apt-get install docker.io=20.10.7-0ubuntu1~20.04.2
...
dpkg: warning: downgrading docker.io from 20.10.7-0ubuntu5~20.04.1 to 20.10.7-0ubuntu1~20.04.2
...

# dockerd --version
Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.2

After the downgrade:

# nvidia-docker run <args> ; echo $?
9fe14058162a8bac4620f1046cf9b6c909d8c3dd03ded17158db524a957562be
0

@somera I got confirmation that the fix will be ported to Jetsons in a few days: https://github.com/NVIDIA/libnvidia-container/issues/148#issuecomment-976814781

Could you please post here https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1950071, including the evidence that you are not using the nvidia runtime; let’s get more info to the Ubuntu devs.

Done!

You might be able to get it from your local apt cache:

# ls -al /var/cache/apt/archives/docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb
-rw-r--r-- 1 root root 36884416 Oct  4 22:33 /var/cache/apt/archives/docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb

If it was removed, you can download it from here:

https://www.dropbox.com/s/keo9i6hx51p5q4b/docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb?dl=0

The checksums can still be found in the Google cache:

https://webcache.googleusercontent.com/search?q=cache:sUZBwHThu7MJ:https://packages.ubuntu.com/focal/amd64/docker.io/download+&cd=1&hl=en&ct=clnk&gl=sg

# sha1sum docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb 
9996131217d5f3da77b82031048938648b5a7634  docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb

# md5sum docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb 
7ab81792216a3de59bb663ae48549f7d  docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb

# sha256sum docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb 
ac7daa552a0a82c91f5bb3777b47907c10bebd0f5a20f6918e56481a6bd3e1c8  docker.io_20.10.7-0ubuntu1~20.04.2_amd64.deb

Same problem here. I updated docker.io and containerd in Ubuntu 18.04 via apt yesterday afternoon.

containerd/bionic-updates 1.5.5-0ubuntu3~18.04.1 arm64 [upgradable from: 1.5.2-0ubuntu1~18.04.2]
docker.io/bionic-updates 20.10.7-0ubuntu5~18.04.2 arm64 [upgradable from: 20.10.7-0ubuntu1~18.04.1]

Downgrading docker, as suggested by @nmtsousa, fixed the issue.

> I think there’s a custom runc binary installed

Thanks @thaJeztah ! I guess I must’ve compiled it myself then forgotten I’d done so, it was using /usr/local/bin/runc.

I am now on v1.0.2-0-g52b36a2, and no longer have the issue. Thanks @cpuguy83 as well.

I would ask that anyone not using nvidia that is experiencing this issue:

If you are using Docker please make sure to update the containerd.io package. Otherwise please make sure you are using runc >= 1.0.

Please run your repro with --runtime=io.containerd.runc.v2 to make extra care that we are using the expected runtime.

If it still reproduces, give us:

1. docker info (if applicable),
2. runc --version,
3. distro + version
4. command you are using that repros the issue
5. Version of libseccomp

Running on a jetson, I still get the issue with

Docker version 20.10.7, build 20.10.7-0ubuntu5~18.04.3

Did anyone resolve this issue on Ubuntu 18 ? Is the fix going to be published to 18 too ?

Most likely the root cause is a nvidia-container-toolkit problem affecting 1.5.1-1 with a proposed fix in 1.6.0~rc.2-1

https://issueexplorer.com/issue/NVIDIA/nvidia-container-runtime/157

It would seem there are two options for temporarily fixing it until 1.6.0 is out:

• Install nvidia-container-toolkit 1.6.0~rc.2-1 https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#setting-up-nvidia-container-toolkit (the experimental section); I did not test this.
• Downgrade to 20.10.7-0ubuntu1~20.04.2 - This works for me; Ubuntu have just removed the package from the distribution, but can still be found online.

Is everyone on this thread using the nvidia runtime?

I have the same problem on ubuntu 20.04 with docker version 20.10.12-0ubuntu2~20.04.1. I tried to downgrade the version to 20.10.7-0ubuntu1~20.04.2 as suggested from this issue. However, this version can no longer be found. apt-get install docker.io=20.10.7-0ubuntu1~20.04.2 Reading package lists... Done Building dependency tree Reading state information... Done E: Version '20.10.7-0ubuntu1~20.04.2' for 'docker.io' was not found Is there any other version i can use?

This should be fixed by nvidia-container-toolkit v1.6.0 (1.7.0 for Jetson… based on the Nvidia issue tracker).

Same problem with ubuntu 20.04 using docker.io but I do not have the ability to downgrade. Switching to docker-ce fixes the issue for me