containerd: container error after upgrade to containerd.io 1.4.6-1

I also have problems after upgrading from 1.4.4-1 to 1.4.6-1.

I get the following error:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:[…] Type:bind Source:/var/lib/docker/volumes/ … /_data Options:[rbind]}: mount destination […] not absolute: unknown.

Downgrading to 1.4.4-1 also helped me immediately.

You are my hero @thaJeztah 🎉

By doing docker inspect cdalvaro/docker-salt-master:latest there are two wrong volumes:

"Volumes": {
  "/home/salt/data/3pfs": {},
  "/home/salt/data/config": {},
  "/home/salt/data/keys": {},
  "/home/salt/data/logs": {},
  "/home/salt/data/srv": {},
  "[": {},
  "]": {}
},

I had a multiple directories VOLUME command without commas. I have separated those directories with commas and it has worked like a charm! (https://github.com/cdalvaro/docker-salt-master/pull/78/commits/7a03130ff544501cc04db967b2fb1060b2822425)

Thank you again for your help!

Yeah, it’s definitely odd. The “source” of that mount (/var/lib/docker/volumes/361b5350807ea92c892fdc87e911040d484c24db5b72b74bf5a865374e416224/_data) looks like it’s an “anonymous” volume (which would be if the container’s image has a VOLUME specified). The error message is weird though, as it would mean that somehow the VOLUME definition in the image that’s used has [ as target. That could be if (e.g.) it was built from a Dockerfile that uses the JSON notation for volumes, something like:

VOLUME ["/some/path/", "/some-other/path/", "/yet/another/path/"]

If that JSON was invalid (e.g., using single quotes instead of double quotes, or a missing comma) during building the image, Docker will fallback to treating that JSON as a literal string (which would explain the error message).

Are you able to do a docker inspect of the cdalvaro/docker-salt-master:latest image and the container that was started?

I was getting:

docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:: Type:bind Source:/var/lib/docker/volumes/9e1805936631f35e4895a18840a0f56c21669099b5967954b176585e9af7a2da/_data Options:[rbind]}: mount destination : not absolute: unknown.

Downgrading as pointed out by @KeyserSoeze worked.

Same thing! Had to downgrade. It fails even for a simple Dockerfile, such as:

FROM ubuntu:18.04

RUN echo asd

docker build -t asd .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:18.04
 ---> 81bcf752ac3d
Step 2/2 : RUN echo asd
 ---> Running in 71fc8a0abb62
OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: no such file or directory: unknown

docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

On Ubuntu 18:04

Tested and it works. Thanks for your explanation.

Upcoming patch release of Docker Desktop will include runc v1.0.0

But in your case looks like there’s a mistake in your Dockerfile;

VOLUME [ /usr/src ]

Is not valid JSON, so will create 3 volumes: [, /usr/src, and ]

Change it to:

VOLUME ["/usr/src"]

(with quotes)

or

VOLUME /usr/src

This issue should be fixed by https://github.com/opencontainers/runc/pull/3004, which relaxed the validation in runc, and is part of runc v1.0.0, which is included in containerd v1.5.3 (and up) and containerd v1.4.7 (and up).