libnetwork: Ubuntu 18.04 + Docker 17.12.1-ce break DNS resolution

I also encountered the same issue after upgrading to Ubuntu 18.04. The root cause of the issue is that in Ubuntu 18.04, the /etc/resolve.conf is controlled by systemd-resolve which is using local address 127.0.0.53 as DNS server. Meanwhile, the docker is also dependent on the DSN in /etc/resolve.conf. In this case since the 127.0.0.53 is only meaningful for the host instead of each Docker container. Thus, Docker choose a fallback way, which set DNS for each container to Google’s public DNS server which is 8.8.8.8.

Although Docker has a --dns to explicitly set external DNS server, it is not a ideal way. I’m not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver: ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

I really don’t think this issue should have been closed- docker is broken out of the box on any corporate network, and in a non-obvious way that will usually require the hunting down of this issue in order to find a work around.

Would it not be possible for docker to detect the presence of nameserver 127.0.0.53 in resolv.conf in a similar way to how systemd-resolve does and use /run/systemd/resolv/resolv.conf (which contains the actual DNS nameserver rather than the local) instead?

We solved this by adding the following to every docker run command. Using this approach we at least didn’t have to modify any system wide files on our file system.

--dns `cat /run/systemd/resolve/resolv.conf | grep nameserver | head -n 1 | awk '{print $2}'`

The fact that we had to do this on all of our Ubuntu 18.04 servers is obnoxious.

@fcrisciani my host system has a /etc/resolv.conf that has a working entry for name resolution. Docker ignores this, and then fails at runtime. That does seem like a bug to me. Can you clarify? It does seem very surprising.

If this is how Docker does and will behave, I guess this is an upstream bug in Ubuntu’s package setup.

Has anyone found a solution for docker-compose yet? I can do it for individual containers, but so far I’m looking at having to dynamically create the docker-compose.yml to set DNS on my dev machine.

I followed @kaneg’s approach:

$ ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 32 Aug  6 12:58 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
$

According to the man page systemd-resolved(8) on my Ubuntu 18.04 system, this is a clean and supported approach (see section “/ETC/RESOLV.CONF” there).

With this setup in place, name resolution inside containers works just fine for me.

Guys

I tried all the solutions but unfortunatelly, none workd for me!.

The best way for me is to force the container to use the same network as the host by using the parameter –network host

Example: $ docker run --name test --network host --rm -id personal/centos6-chef 854d161cd8a16040830f7c58bc541b0d992dc9567d8804edc745ecebb6738cb8

$ docker exec -it test bash [root@CO-IT01776 /]# ping google.com PING google.com (216.58.217.206) 56(84) bytes of data. 64 bytes from lax17s05-in-f206.1e100.net (216.58.217.206): icmp_seq=1 ttl=49 time=120 ms 64 bytes from lax17s05-in-f206.1e100.net (216.58.217.206): icmp_seq=2 ttl=49 time=120 ms

With docker-compose.yml

version: '3.2'
services:
  remy:
    image: personal/centos6-chef:latest
    environment:
    network_mode: host
    privileged: true
    volumes:

Greetings from Colombia Thanks

I definitely agree that this should be treated like a bug. The solution @kaneg suggests unfortunately doesn’t work on my system (Ubuntu 18.04, Docker 18.06.0-ce). This is my output:

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 32 Aug 28 10:06 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

however the container still have the incorrect configuration when started with docker-compose.

The only ‘workaround’ I’ve found is to manually append the contents of /etc/resolv.conf from the host to the same file in the container. As others have pointed out, ADD or COPY on build doesn’t work. Perhaps the file could be populated with the correct dns info with an entrypoint script, but honestly I’ve already wasted a few hours trying to get this to work and I don’t restart my containers too often.

I agree there is an issue here somewhere…

I have docker 18.03.0-ce running on Ubuntu 16.04 with custom DNS entries in /etc/resolv.conf, e.g.: nameserver 172.99.1.7, and in this environment if I look at /etc/resolv.conf inside a container I see the same thing: nameserver 172.99.1.7

Now on a new machine running docker 18.05.0-ce on Ubuntu 18.04 with the same DNS setup but configured with netplan (which is part of 18.04 changes). The /etc/resolv.conf file should not be edited by hand and uses systemd-resolved - it has an internal nameserver 127.0.0.53 entry in /etc/resolv.conf. Now inside the container I don’t see this entry copied but instead get the default google ones (8.8.8.8, 8.8.4.4).

Clearly Ubuntu is different w.r.t. DNS in 18.04, but I haven’t confirmed whether Docker is also behaving differently, or just defaults to Google DNS because it somehow knows that 127.0.0.53 is not going to work inside the container.

My workaround for now is to use the --dns option when starting containers to manually specify the hosts, but I would rather the DNS inheritance from the host continued to work!

Documentation ref: https://docs.docker.com/config/containers/container-networking/#dns-services “By default, a container inherits the DNS settings of the Docker daemon, including the /etc/hosts and /etc/resolv.conf. You can override these settings on a per-container basis”

Related issue? https://github.com/docker/libnetwork/issues/2068

As this thread is referenced by Google, let me provide “how-to” on resolving the issue with the docker builds on Ubuntu 18.04.

• find out the DNS servers with command (your IP addressed will be different):

$ nmcli dev show | grep DNS
IP4.DNS[1]:                             10.11.12.13
IP4.DNS[2]:                             10.11.12.14

• register these IPs with the /etc/resolv.conf:

sudo -s
echo "nameserver 10.11.12.13" >> /etc/resolvconf/resolv.conf.d/tail
echo "nameserver 10.11.12.14" >> /etc/resolvconf/resolv.conf.d/tail

• reboot the OS and check whether the IPs have been registered:

cat /etc/resolv.conf   

I’m not sure if there is any elegant way to resolve the issue. For now, I use below workaround by disabling systemd as default DSN resolver: ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Just a warning when I ran this command I ended up losing internet access for the whole machine.

@mushkevych What OS are you using? in 18.04 there’s no resolv.conf.d/tail out of the box.

I still have the issue on 18.09.0, build 4d60db4. I bypass it by adding a dns list.

# using OpenDNS
dns:
  - "208.67.222.222"
  - "208.67.220.220"

@mushkevych It didn’t solve it for me, do you have perhaps resolvconf installed? It does not come shipped with 18.04 neither.