moby: Containers cannot access internet (outbound)

For anyone running into this, rebooting the OS fixed it.

Has anyone managed to solve this any other way?

I’ve tried restarting the Docker service, rebooting the OS, and forcing a bridge rebuild. No success so far.

I’m running Ubuntu 18.10 (Pop!_OS 18.10) and Docker version 18.06.1-ce, build e68fc7a.

Solved it for me:

Since systemd version >=219 ip forwarding (net.ipv4.conf.interfaceName.forwarding) is off by default. To activate I had to place the option IPForward=true in the respective systemd network config file:

[Network]
...
IPForward=true

After restart of systemd-networkd (systemctl restart systemd-networkd) everything works as expected.

Reference: https://docs.docker.com/v17.09/engine/installation/linux/linux-postinstall/#ip-forwarding-problems

Having the same issue on one frech installed CentOS 7.7 minimal (Hetzner Server). Tested with docker 18.09.9 and latest 19.03.3. No custom configuration for CentOS and docker.

I tried basically everything, but no luck. Very very strange, since I manage dozens of servers with the same stack and configuration (ansible provisioned), not having this problem??

On this server (Type EX 42 - https://www.hetzner.de/dedicated-rootserver/ex42) the problem is reproducible:

1. Minimal CentOS 7.6 install (by Hetzner installimage)
2. yum update (also tried without it)
3. docker installation: yum install -y yum-utils yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum install docker-ce docker-ce-cli containerd.io systemctl start docker
4. Tests:

docker run -it --rm alpine ping 1.1.1.1 _> does NOT work

docker run -it --rm --net=host alpine ping 1.1.1.1 _> DOES work

Additional info

uname -a 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core)

docker version Client: Docker Engine - Community Version: 19.03.3 API version: 1.40 Go version: go1.12.10 Git commit: a872fc2f86 Built: Tue Oct 8 00:58:10 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Community Engine: Version: 19.03.3 API version: 1.40 (minimum version 1.12) Go version: go1.12.10 Git commit: a872fc2f86 Built: Tue Oct 8 00:56:46 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683

Any ideas? Thanks!

try

systemctl stop firewalld

or

firewall-cmd --permanent --zone=trusted --change-interface=docker0 firewall-cmd --reload systemctl restart docker

After A LOT of time trying to solve this issue I think I finally found a solution in an ebook I found on the internet.

Here’s the steps I took to solve the issue:

$ sudo service docker stop
$ sudo nano /etc/default/docker

Edit the file and uncomment the DOCKER_OPTS option modifying it to somthing like this: DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --iptables=false --ip-forward=false"

$ sudo su
$ echo 1 > /proc/sys/net/ipv4/ip_forward
// Add the IP range you want to have access to the internet
$ iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE
$ service docker restart
$ exit

Now hopefully your containers have access to the internet again.

I hope to help someone 😃

I’ve had a similar networking problem and found a solution…

Under an hypervisor, I had made a template out of a plain Docker VM. I then cloned it into 3 different hosts, clearly sharing the exact same config and data, at least at first boot.

From there on, I started having containers unable to reach the internet.

The solution? Wipe all configs and data.

systemctl stop docker
rm -fr /var/lib/docker
rm -fr /etc/docker
systemctl start docker

Seems like Docker re-configures itself in that case. The next thing I’ve realized, my containers started working flawlessly.

Enabling ip forwarding (which had become suddenly disabled in the host) worked for me. sudo sysctl net.ipv4.ip_forward=1

I used to stop nftables before starting docker :

systemctl stop nftables systemctl start docker

It stopped working, containers couldn’t reach internet.

Disabling nftables and rebooting fixed the issue :

systemctl disable nftables reboot systemctl start docker

Will try downgrading nftables.

for me Rebooting the OS worked.

After three years of using docker without problems, this issue is affecting me today. The only two new things at the time it showed up are (1) used a debian container for the first time, (2) added my user to the docker group and began using docker without sudo.

Attempted fixes which did not work:

1. sudo systemctl restart docker

2. Toggle --iptables dockerd switch between true and false, restarting docker service between toggles.

3. Adding VIF using sudo brctl addif docker0 <vif>.

4. Forcing rebuild of bridge

sudo systemctl stop docker
sudo pkill docker
sudo iptables -t nat -F
sudo ifconfig docker0 down
sudo brctl delbr docker0
sudo systemctl start docker

I am using docker version 17.12.1-ce, build 7390fc6 on Ubuntu 17.10.

What worked for me is loading the br_netfilter kernel module:

lsmod | grep br_netfilter
modprobe br_netfilter

This was on a CentOS 7 node, where I had to upgrade the kernel before I could load the kernel module.

Another possible solution is to run the docker0 interface in promiscuous mode:

# Check if it's in promisc mode (Check for P in the Flg column of the output)
netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0   1500  3058281      0      0 0       5478012      0      0      0 BMU
eth0      1400 19935201      0      0 0      19519868      0      0      0 BMRU
lo       65536  2372033      0      0 0       2372033      0      0      0 LRU

ip link set dev docker0 promisc on

I’m also seeing this issue on Ubuntu 16 & 18 LTS:

 ▲ ~ docker run --rm alpine ping -c 2 www.google.com
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
9d48c3bd43c5: Pull complete
Digest: sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb
Status: Downloaded newer image for alpine:latest
ping: bad address 'www.google.com'

Same things with different image:

 ▲ ~ docker run --network bridge --rm -it ubuntu:trusty bash                                                                                     master 3d :: ⬡
root@d846d33bcbd3:/# ping google.com
ping: unknown host google.com
root@d846d33bcbd3:/#

Docker Version:

 △ ~ docker version                                                                                                                                                                       12s
Client:
 Version:           18.09.7
 API version:       1.39
 Go version:        go1.10.1
 Git commit:        2d0083d
 Built:             Fri Aug 16 14:20:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.7
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.1
  Git commit:       2d0083d
  Built:            Wed Aug 14 19:41:23 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Things I’ve attempted to fix this issue with no luck:

1. I’ve removed docker0 and restarted docker (daemon)
2. Ensured that the allocated ip addresses for docker don’t conflict with any DNS IPs
3. Upgraded from Ubuntu 16.04 to Ubuntu 18.04
4. Toggled the --iptables flag
5. Made the DNS servers 8.8.8.8 and 8.8.4.4 explicitly in /etc/docker/daemon.json
6. flushed iptables iptables -F
7. Ensured net.ipv4.ip_forward is set to 1

Note, this is happening out of the box. The first thing I did after I setup my new Ubuntu 18 OS was install docker / docker-compose and ran docker run --rm alpine ping -c 2 www.google.com.

Also wanted to point out that using the host network driver allows me to make any requests to the outside world. But, using the default bridge or a custom network (with the bridge driver) results in no access to the internet.

Really hoping this can be resolved 🙏

Lost my whole day to this. A reboot (Win 10) worked in the end. Thanks @expz

reboot helped.

For me, this problem occurred because I was using nftables and had installed my own inet filter table for both ipv4 and ipv6. This table default-dropped traffic from docker. Now you’d think that docker installs rules to get around this default drop through iptables-nft, but that only affects the ip filter table. Both tables need to accept the traffic.

After A LOT of time trying to solve this issue I think I finally found a solution in an ebook I found on the internet.

Here’s the steps I took to solve the issue:

$ sudo service docker stop
$ sudo nano /etc/default/docker

Edit the file and uncomment the DOCKER_OPTS option modifying it to somthing like this: DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --iptables=false --ip-forward=false"

$ sudo su
$ echo 1 > /proc/sys/net/ipv4/ip_forward
// Add the IP range you want to have access to the internet
$ iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE
$ service docker restart
$ exit

Now hopefully your containers have access to the internet again.

I hope to help someone 😃

Thank you!! Your solution works for me!

I’m sorry to say this but it really sounds that you are seeing this issue because you are using non-supported configuration(s).

@trinhpham combination of Windows Server and CE version of Docker is not supported. Please use either Win 10 with Docker CE (if you need Linux support) or Win Srv with enterprise edition.

@DMeechan I have ever heard about Pop!_OS but I guess that it causes your problem. Please, test with normal version of Ubuntu and latest Docker-CE version

@prakashs-treselle I think that AWS Linux 2 is not supported Docker host OS. How ever you can run it inside of Docker.

@jobinau please check that you don’t have any special iptables rules. Docker uses iptables to provide networking for containers.

Here is example from Ubuntu box with standard config and latest version of Docker:

$ docker run --rm alpine ping -c 2 www.google.com
PING www.google.com (216.58.207.228): 56 data bytes
64 bytes from 216.58.207.228: seq=0 ttl=54 time=21.100 ms
64 bytes from 216.58.207.228: seq=1 ttl=54 time=20.647 ms

--- www.google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 20.647/20.873/21.100 ms

And here is same from Windows Server 2019 with latest Docker version on Azure:

PS C:\> docker run --rm mcr.microsoft.com/windows/nanoserver:1809 ping -n 2 www.google.com

Pinging www.google.com [216.58.211.100] with 32 bytes of data:
Reply from 216.58.211.100: bytes=32 time=1ms TTL=50
Reply from 216.58.211.100: bytes=32 time=1ms TTL=50

Ping statistics for 216.58.211.100:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 1ms, Average = 1ms

In my case, the reboot of the OS probably solved the problem because it killed the VPN connection and restarted the docker service. There is some conflict between my VPN connection and the docker networking setup. You could also try the four methods I listed.