moby: Docker DNS not resolving certain addresses

Description

Can’t resolve certain domains (security.debian.org) from inside a container running on a Docker host in swarm mode.

The exact same nslookup works on the Docker host itself.

Steps to reproduce the issue:

1. docker exec -it <container ID> /bin/sh
2. Run the following:

/go # nslookup security.debian.org
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'security.debian.org': Try again

Describe the results you received: I get nslookup: can't resolve 'security.debian.org': Try again.

Describe the results you expected:

Server:		168.63.129.16
Address:	168.63.129.16#53

Non-authoritative answer:
Name:	security.debian.org
Address: 195.20.242.89
Name:	security.debian.org
Address: 212.211.132.32
Name:	security.debian.org
Address: 212.211.132.250
security.debian.org	rdata_46 = A 8 3 300 20170113080228 20161214080228 26616 security.debian.org. YHXIBebtdCUWZrUQYH/7YxaTCC5sAc75AjqG62kVtGinDWISq00RJJnr arErmSYjkwnOI9CLuTHTBHITojwziLMb32ShnSLuKLv9y79tBO9y4von NqzKb0jL0ra7+NFhG2rPoqk4wro+w8h5THtlV7hJjWcBO0+4keMaIrz7 RIIt3LsdqbhODPifksnU29KqiX4q3+KLCeW191HXnFAzx7/wuCKeGLLR N1LV8E60VkrgWcls2mY8SrC5RolD0/GV
security.debian.org	rdata_46 = A 8 3 300 20170113080228 20161214080228 27264 security.debian.org. Ipaq1Mc+p8yhZ8qshhlWzj8QOLysgVv4+IjW1cThoMPkAvPClrzxyFw8 ZQ9ZB4ZKpuzc5tYeaT1QaE5VtIRyQYOIrJmjO8QL9sK0hav60aa/9XAD IsB3bo/WqRhU+czbijD2Thmou8G2h62TQ3sANIdi7iphpPLfzsBGvYLL 4x1VoDzcz8164NZyIqjz2prYRibaSWOsBrVDaWeTAFJ3Lng/FW8NXTtK GUwkYUopfVkkwaC4672qFYWDaJrs5dvZ

Additional information you deem important (e.g. issue happens only occasionally): It happens reliably. The container uses the Docker DNS.

tcpdump on the host shows that Docker communicates with the Azure DNS but the DNS replies do not end up in the container for some reason.

root@prodmgr1:~# tcpdump -i eth0 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:39:24.028940 IP 172.16.0.4.45116 > 168.63.129.16.domain: 61501+ AAAA? security.debian.org. (37)
11:39:24.029140 IP 172.16.0.4.34565 > 168.63.129.16.domain: 60901+ A? security.debian.org. (37)
11:39:24.029941 IP 172.16.0.4.46180 > 168.63.129.16.domain: 36328+ PTR? 16.129.63.168.in-addr.arpa. (44)
11:39:24.113399 IP 168.63.129.16.domain > 172.16.0.4.45116: 61501 4/0/0 AAAA 2001:a78:5:1:216:35ff:fe7f:6ceb, AAAA 2001:a78:5:0:216:35ff:fe7f:be4f, RRSIG, RRSIG (579)
11:39:24.113472 IP 168.63.129.16.domain > 172.16.0.4.34565: 60901 5/0/0 A 212.211.132.250, A 212.211.132.32, A 195.20.242.89, RRSIG, RRSIG (571)
11:39:24.113560 IP 168.63.129.16.domain > 172.16.0.4.46180: 36328 NXDomain 0/1/0 (112)
11:39:24.114805 IP 172.16.0.4.56977 > 168.63.129.16.domain: 854+ PTR? 4.0.16.172.in-addr.arpa. (41)
11:39:26.531518 IP 172.16.0.4.33028 > 168.63.129.16.domain: 61501+ AAAA? security.debian.org. (37)
11:39:26.531531 IP 172.16.0.4.58915 > 168.63.129.16.domain: 60901+ A? security.debian.org. (37)
11:39:26.535494 IP 168.63.129.16.domain > 172.16.0.4.33028: 61501 4/0/0 AAAA 2001:a78:5:0:216:35ff:fe7f:be4f, AAAA 2001:a78:5:1:216:35ff:fe7f:6ceb, RRSIG, RRSIG (579)
11:39:26.535554 IP 168.63.129.16.domain > 172.16.0.4.58915: 60901 5/0/0 A 212.211.132.32, A 195.20.242.89, A 212.211.132.250, RRSIG, RRSIG (571)

Output of docker version:

Client:
 Version:      1.12.4
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   1564f02
 Built:        Tue Dec 13 00:08:34 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.4
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   1564f02
 Built:        Tue Dec 13 00:08:34 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 38
 Running: 2
 Paused: 0
 Stopped: 36
Images: 11
Server Version: 1.12.4
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 124
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay host bridge null
Swarm: active
 NodeID: 04fskwcb5p93m448s03pbelvd
 Is Manager: true
 ClusterID: d2l1ktp0920uyqey500caiezj
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.16.0.4
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-53-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: prodmgr1
ID: SJG2:QO7A:DNZ6:LXBE:AC3M:45HR:TWR6:YNCQ:4Q66:62DE:KXZK:WLDN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: econdockerplayground
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.): Running on Azure.