moby: upgrade docker-18.09.2-ce , shim.sock: bind: address already in use: unknown

Description

Steps to reproduce the issue: 1.upgrade docker from 18.03.1-ce to 18.09.2-ce

some container show failed to bind address I have “ps -ef|grep mysqld” and ss -nlp|grep 3306 ,cleaned process
docker start xxxx , it still show the some error

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:47:25 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 22
 Running: 20
 Paused: 0
 Stopped: 2
Images: 41
Server Version: 18.09.2
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.152-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.67GiB
Name: dev-rac-agent-01
ID: FNDH:GCFF:P7GX:7SU3:F3JY:2NSS:DSGX:7WOT:HDA4:TQDC:LT5T:SPDF
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.): CentOS Linux release 7.5.1804 (Core) on xen

anyone has this issue ?

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 8
Comments: 17 (2 by maintainers)

Most upvoted comments

in my case this work for me: sudo apt update && sudo apt upgrade -y && sudo shutdown -r now after host has restarted - all containers is UP without re-creations and downgrade.

MindPhaser34 on Jul 11, 2019

The solution described by @suutari worked for me. Downgrade to the version suggested by @Max95Cohen. Remember to kill every process with “container” in the name before downgrade:

ps auxf | grep container
sudo kill pid1 pid2...

It’s important to restart all the containers with the downgraded version of docker.io before stopping them.

daquinoaldo on Jun 4, 2019

Hi, there. My system is Ubuntu 18.04.

I do this: sudo apt-cache policy docker.io

Output:

docker.io:
  Installed: 18.09.5-0ubuntu1~18.04.2
  Candidate: 18.09.5-0ubuntu1~18.04.2
  Version table:
 *** 18.09.5-0ubuntu1~18.04.2 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     18.06.1-0ubuntu1.2~18.04.1 500
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     17.12.1-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

sudo apt install docker.io=18.06.1-0ubuntu1.2~18.04.1 sudo apt autoremove

Then restart my docker container. This is work to me.

Max95Cohen on Jun 4, 2019

This happened to me too on Ubuntu 18.04 when upgrading Docker from 18.06.1-0ubuntu1.2~18.04.1 to 18.09.5-0ubuntu1~18.04.2. I was able to start the failing containers with following steps:

Kill the container processes by hand (they were left running for some reason, even though docker shows them as stopped) (If you’re unable to find them in the ps auxf output, maybe simply rebooting might help.)
Downgrade the docker.io package to the previously used version
Restart Docker: systemctl restart docker.service
Start the failing containers with docker start CONTAINER_ID. Had to do this twice though, since for some reason the first start returned Error response from daemon: id already in use error, but the second try succeeded.
Stop the failing containers with docker stop CONTAINER_ID
Upgrade back to the newest version
Restart docker service again
Now the failing containers were able to start just fine with docker start CONTAINER_ID

suutari on May 26, 2019

I experience a similar issue on version 18.09.3

835400486f5a        60b7495196e2        "/app/core-vitals -c…"   7 weeks ago         Exited (1) 4 days ago                       core-vitals-25
Mon 06 May 2019 11:02:04 AM
$ docker start 835400486f5a
Error response from daemon: failed to listen to abstract unix socket "/containerd-shim/moby/835400486f5a9eb0466c6fd1e7a53bae969aaa56dc23b9928179f5f7d5c4988e/shim.sock": listen unix /containerd-shim/moby/835400486f5a9eb0466c6fd1e7a53bae969aaa56dc23b9928179f5f7d5c4988e/shim.sock: bind: address already in use: unknown
Error: failed to start containers: 835400486f5a

Deleting the container and recreating seems to resolve the issue

elfgoh on May 6, 2019

Same trouble after upgrading from 18.06. Same containers started successfully, other don’t. Ubuntu 18.04 on baremetal edited: after removing old containers by ‘docker rm’ new is starting successfully

sotona- on Feb 20, 2019

You can avoid the reboot

Find the ID of your failed container

docker ps -a
 005ac6424215        nginx:alpine        "/docker-entrypoint.…"   18 minutes ago      Exited (128) 16 minutes ago                       nginx-test-02

find the PID of your failed container

[root@ns7dev9 ~]# ps aux | grep 005ac6424215
root     32627  0.0  0.4 108808  7720 ?        Sl   20:06   0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/005ac6424215a21b791c54311d6df18b124e3a3cde5cf4c82c0eea91a0a1e539 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc

kill the PID of the failed container [root@ns7dev9 ~]# kill -9 32627

start the container

[root@ns7dev9 ~]# docker start 005ac6424215 005ac6424215

stephdl on Jun 6, 2020

After a 18.06 -> 19.02 upgrade on RHEL 7 using the upstream docker-ce packages, containers fail to (re)start until they are removed/pruned:

Sep 11 14:13:22 XXX dockerd[1430]: time="2019-09-11T14:13:22.980144454Z" level=error msg="Handler for POST /v1.40/containers/61e8d9bc175fa4db21be994769d7e209ea8baba6bdec4aae95a96a604ed66f83/start returned error: failed to listen to abstract unix socket \"/containerd-shim/moby/61e8d9bc175fa4db21be994769d7e209ea8baba6bdec4aae95a96a604ed66f83/shim.sock\": listen unix \x00/containerd-shim/moby/61e8d9bc175fa4db21be994769d7e209ea8baba6bdec4aae95a96a604ed66f83/shim.sock: bind: address already in use: unknown"

After re-creating all of the containers, confirm that there are a bunch of old docker-containerd-shim processes left hanging around with the old container IDs:

$ ps auxf | grep containerd-shim
root      9895  0.0  0.0   8896  2968 ?        Sl   Sep11   0:02 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/338d53969f55adbfad0ea0f4d8f4246516e1a674c46040f25aad60cfe7da2223 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
...
root      9220  0.1  0.0 108744  4736 ?        Sl   07:01   0:01  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/5b77ed414614b1de52ecb25283273c657cafbabfc0f77f02faaa9452151fab8b -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
...

I assume this is related to the new containerd.io split in the 18.09 packaging.

Looking at systemctl status, these old docker-containerd-shim processes are under the systemd docker.service cgroup, and the new containerd-shim processes are under the new containerd.service cgroup:

           └─system.slice
             ├─containerd.service
             │ ├─  395 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/cc5f27da484d41590c172a4b815400abdd46f6645efa7b6c752be624fd0562fc -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
             │ ├─  ...
             │ ├─ 1428 /usr/bin/containerd
             │ ├─  ...
             ├─docker.service
             │ ├─ 1430 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
             │ ├─ 9895 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/338d53969f55adbfad0ea0f4d8f4246516e1a674c46040f25aad60cfe7da2223 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
             │ ├─ ...

I’m guessing that due to KillMode=process, a systemctl stop/restart docker will leave those old docker-containerd-shim processes running, and a reboot is indeed the easiest way to get these cleaned up?

SpComb on Sep 12, 2019

Same issue after upgrading to version 18.09.5 on a Ubuntu 18.04

Error response from daemon: failed to listen to abstract unix socket "/containerd-shim/moby/[...]/shim.sock": listen unix /containerd-shim/moby/[...]/shim.sock: bind: address already in use: unknown

All my running containers at the time of update (4 of them) were affected and needed to be recreated. My stopped containers (2 of them) on the other hand were not affected and I could still start them up normally after the update

fimbulwint on May 23, 2019