moby: Docker 20.10.6: all containers stopped and cannot start if ipv6 is disabled on host

Being fixed in https://github.com/moby/moby/pull/42322

Workaround: docker run -p 0.0.0.0:80:80 instead of docker run -p 80:80.

Proper fix is discussed in https://github.com/docker/for-linux/issues/1233#issuecomment-819047829

I am also experiencing this issue on a fresh Centos 7 install with docker-ce 20.10.6 with ipv6 disabled:

docker run  -p 5432:5432 <image-name>

docker: Error response from daemon: driver failed programming external connectivity on endpoint <snip>: Error starting userland proxy: listen tcp6 [::]:5432: socket: address family not supported by protocol.

likely related to https://github.com/moby/moby/pull/42205

Seeing the same thing here. Docker is basically broken for any systems with IPv6 disabled.

Downgrade (Ubuntu 18):

# IPv6 disabled in kernel (don't forget sudo update-grub and sudo reboot):
$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 maybe-ubiquity"
GRUB_CMDLINE_LINUX="ipv6.disable=1"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

$ sudo cat /etc/docker/daemon.json
{
    "default-address-pools": [
        {"base": "10.0.0.0/8", "size": 24}
    ],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "ipv6": false,
    "log-driver": "json-file",
    "log-opts": {
        "max-file": "100",
        "max-size": "100m"
    }
}

$ sudo apt-cache madison docker-ce
 docker-ce | 5:20.10.6~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:20.10.5~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:20.10.4~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:20.10.3~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:20.10.2~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:20.10.1~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:20.10.0~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.15~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.14~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.13~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.12~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.11~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.10~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.9~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.8~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.7~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.6~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.5~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.4~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.3~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.2~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.1~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:19.03.0~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.9~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.8~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.7~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.6~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.5~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.4~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.3~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.2~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.1~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 5:18.09.0~3-0~ubuntu-bionic | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 18.06.3~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 18.06.2~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 18.06.1~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 18.06.0~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
 docker-ce | 18.03.1~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages

# Block 20.10.*:
$ sudo cat /etc/apt/preferences.d/docker  # create if not existent
Package: docker-ce*
Pin: version 5:20.10.*
Pin-Priority: -10

$ sudo apt update
$ sudo systemctl stop docker
$ sudo apt remove docker-ce*
$ sudo apt install docker-ce  # some minor errors - can be ignored?!
$ sudo systemctl start docker

The same for Puppet script file:

#!/bin/bash

exec &>> /tmp/downgrade_docker.log

/usr/bin/docker --version | /bin/grep '20\.10' && [ -e /etc/apt/preferences.d/docker ] && {
    /usr/bin/apt-get --yes update || true
    /bin/systemctl stop docker || true
    /usr/bin/apt-get --yes remove docker-ce* || true
    /usr/bin/apt-get --yes install docker-ce || true
    /bin/systemctl start docker || true
}

true

That shit effected a lot of systems here. Because we tried setting "userland-proxy": false before. Not good. This leads to the fact that some containers can’t reach other VMs/ports. Can someone please inform which versions are (and will be) affected/OK (again)? Thanks.

Workaround: docker run -p 0.0.0.0:80:80 instead of docker run -p 80:80.

Proper fix is discussed in docker/for-linux#1233 (comment)

That is a painful solution as it requires updating the startup scripts/compose files for all containers!

@pzi123 We are all volunteers on this project. By all means please make sure to test master builds to ensure it works for you.

Workarounds are to not disable ipv6 on boot, hardcode a default address to listen on instead of 0.0.0.0, or downgrade to 20.10.5.

For good measure I have also blocked the 20.10.6 version of docker-ce-cli and docker-ce-rootless-extras:

user@server:/etc/apt/preferences.d$ ls -lah
total 20
drwxr-xr-x 2 root root 4096 Apr 15 10:55 ./
drwxr-xr-x 7 root root 4096 Apr 15 10:32 ../
-rw-r--r-- 1 root root   77 Apr 15 10:51 docker-ce
-rw-r--r-- 1 root root   81 Apr 15 10:53 docker-ce-cli
-rw-r--r-- 1 root root   93 Apr 15 10:55 docker-ce-rootless-extras

user@server:/etc/apt/preferences.d$ for tt in $(find . -type f); do echo $tt; cat $tt; echo ; done
./docker-ce
Package: docker-ce
Pin: version 5:20.10.6~3-0~ubuntu-focal
Pin-Priority: -10

./docker-ce-rootless-extras
Package: docker-ce-rootless-extras
Pin: version 5:20.10.6~3-0~ubuntu-focal
Pin-Priority: -10

./docker-ce-cli
Package: docker-ce-cli
Pin: version 5:20.10.6~3-0~ubuntu-focal
Pin-Priority: -10

Doing a downgrade to version 20.10.5 is a viable workaround.

It looks like an official release with the fix was pushed out today: 20.10.7 release notes

putting explicitly the ipv4 interface you want to use, fixes the issues both in docker as in docker-compose. like

ports: - “0.0.0.0:X:Y/tcp”

At what point does the change get reverted and a new release cut?

This broke ECS.

Can confirm the same issue on CentOS Linux release 8.3.2011 with ipv6 disabled and docker-ce 20.10.6-3.el8

When using the “ports:” directive in a docker compose file it fails to start. Without specifying port mapping it starts fine.

Automate everything 😛

Well, of course, that is implied. 😃 But, the idea would be to unhold after the docker team fixes the issue.

Workaround: docker run -p 0.0.0.0:80:80 instead of docker run -p 80:80. Proper fix is discussed in docker/for-linux#1233 (comment)

That is a painful solution as it requires updating the startup scripts/compose files for all containers!

downgrade is much easier

+1 here on Red Hat 7.9.

Downgrading from docker-ce.x86_64 3:20.10.6-3.el7 to docker-ce.x86_64 3:20.10.5-3.el7 seems to work for now.

Just to create a link: At https://forums.docker.com/t/ipv6-disabled-on-my-computer-but-docker-network-seems-looking-for-it/107299/16 more people are discussing intermediate solutions etc.

Can confirm the problem after update today. I’m on: 5.8.0-50-generic #56~20.04.1-Ubuntu SMP Mon Apr 12 21:46:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ipv6 disabled via Grub.

And because I like number 19 more than 20 I pinned to 19.03.15 and I’m able to use docker-compose up again without problems.