moby: docker 1.10 embedded DNS doesn't work as documented

Description of problem:

I installed Docker engine 1.10 from the repository in a CentOS 7 VM, and followed the steps here: https://docs.docker.com/engine/userguide/networking/work-with-networks/

However this step doesn’t work: / # ping -w 4 container3, in fact name resolution doesn’t work at all with user defined networks, and I get some iptables errors from docker when creating the user-defined network.

What am I doing wrong?

docker version:

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

docker info:

Containers: 13
 Running: 4
 Paused: 0
 Stopped: 9
Images: 8
Server Version: 1.10.0
Storage Driver: devicemapper
 Pool Name: docker-253:0-18908918-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/vg-docker/data
 Metadata file: /dev/vg-docker/metadata
 Data Space Used: 625.3 MB
 Data Space Total: 10.74 GB
 Data Space Available: 10.11 GB
 Metadata Space Used: 1.282 MB
 Metadata Space Total: 1.074 GB
 Metadata Space Available: 1.072 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 7
Total Memory: 7.639 GiB
Name: localhost.localdomain
ID: PVJM:C6JB:LAFE:PURU:4HQI:23E6:DS22:7J2O:3XT3:VTZJ:A33N:NYZC
Debug mode (server): true
 File Descriptors: 54
 Goroutines: 85
 System Time: 2016-02-05T15:49:00.057923818+02:00
 EventsListeners: 0
 Init SHA1: 6ad5ded80380758740d165409e0c26db46e05a9f
 Init Path: /usr/libexec/docker/dockerinit
 Docker Root Dir: /var/lib/docker
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
x86_64 x86_64 GNU/Linux
[root@localhost ~]# getenforce
Enforcing

Environment details: Virtual:

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 

How reproducible: always

Steps to Reproduce:

1. docker run -itd --name=container1 busybox
2. docker run -itd --name=container2 busybox
3. docker network create -d bridge --subnet 172.25.0.0/16 isolated_nw
4. docker run --net=isolated_nw --ip=172.25.3.3 -itd --name=container3 busybox
5. docker attach container2
6. ping -w 4 container3
7. ping -w 4 google.com

Actual Results:

ping: bad address 'container3'
ping: bad address 'google.com'

Expected Results: Able to ping container3 and google

Additional info: The DNS server is reachable from inside the container with ping, it just doesn’t seem to answer DNS requests properly:

/ # cat /etc/resolv.conf 
nameserver 127.0.0.11
options ndots:0
/ # ping -w 4 127.0.0.11
PING 127.0.0.11 (127.0.0.11): 56 data bytes
64 bytes from 127.0.0.11: seq=0 ttl=64 time=0.141 ms
64 bytes from 127.0.0.11: seq=1 ttl=64 time=0.066 ms
64 bytes from 127.0.0.11: seq=2 ttl=64 time=0.127 ms
64 bytes from 127.0.0.11: seq=3 ttl=64 time=0.059 ms

--- 127.0.0.11 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.059/0.098/0.141 ms

I enabled -D and I get some iptables errors from docker in the journal:

Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.739603897+02:00" level=debug msg="Calling POST /v1.22/networks/create"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.739656325+02:00" level=debug msg="POST /v1.22/networks/create"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.739757830+02:00" level=debug msg="form data: {\"CheckDuplicate\":true,\"Driver\":\"bridge\",\"IPAM\":{\"Config\":[{\"Subnet\":\"172.25.0.0/16\"}],\"Driver\":\"default\",\"Options\":{}},\"Internal\":false,\"Name\":\"isolated_nw\",\"Options\":{}}"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.740009686+02:00" level=debug msg="Allocating IPv4 pools for network isolated_nw (b0f784c11c0d2301b7d0f1bc5dd9f4500b823425e61687fd7c21dd9319fb5146)"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.740035294+02:00" level=debug msg="RequestPool(LocalDefault, 172.25.0.0/16, , map[], false)"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.772306044+02:00" level=debug msg="RequestAddress(LocalDefault/172.25.0.0/16, <nil>, map[RequestAddressType:com.docker.network.gateway])"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.787453719+02:00" level=debug msg="Setting bridge mac address to 02:42:32:28:d2:1b"
Feb 05 15:45:49 localhost.localdomain NetworkManager[793]: <info>  (br-b0f784c11c0d): new Bridge device (carrier: OFF, driver: 'bridge', ifindex: 152)
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.787602151+02:00" level=debug msg="Assigning address to bridge interface br-b0f784c11c0d: 172.25.0.1/16"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.787968759+02:00" level=debug msg="Firewalld passthrough: ipv4, [-t nat -C POSTROUTING -s 172.25.0.0/16 ! -o br-b0f784c11c0d -j MASQUERADE]"
Feb 05 15:45:49 localhost.localdomain firewalld[711]: 2016-02-05 15:45:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.25.0.0/16 ! -o br-b0f784c11c0d -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.803047278+02:00" level=debug msg="Firewalld passthrough: ipv4, [-t nat -I POSTROUTING -s 172.25.0.0/16 ! -o br-b0f784c11c0d -j MASQUERADE]"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.813140080+02:00" level=debug msg="Firewalld passthrough: ipv4, [-t nat -C DOCKER -i br-b0f784c11c0d -j RETURN]"
Feb 05 15:45:49 localhost.localdomain firewalld[711]: 2016-02-05 15:45:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER -i br-b0f784c11c0d -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.826083377+02:00" level=debug msg="Firewalld passthrough: ipv4, [-t nat -I DOCKER -i br-b0f784c11c0d -j RETURN]"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.835311082+02:00" level=debug msg="Firewalld passthrough: ipv4, [-D FORWARD -i br-b0f784c11c0d -o br-b0f784c11c0d -j DROP]"
Feb 05 15:45:49 localhost.localdomain firewalld[711]: 2016-02-05 15:45:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -D FORWARD -i br-b0f784c11c0d -o br-b0f784c11c0d -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.844823744+02:00" level=debug msg="Firewalld passthrough: ipv4, [-t filter -C FORWARD -i br-b0f784c11c0d -o br-b0f784c11c0d -j ACCEPT]"
Feb 05 15:45:49 localhost.localdomain firewalld[711]: 2016-02-05 15:45:49 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C FORWARD -i br-b0f784c11c0d -o br-b0f784c11c0d -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.856573443+02:00" level=debug msg="Firewalld passthrough: ipv4, [-I FORWARD -i br-b0f784c11c0d -o br-b0f784c11c0d -j ACCEPT]"
Feb 05 15:45:49 localhost.localdomain docker[4175]: time="2016-02-05T15:45:49.865931993+02:00" level=debug msg="Firewalld passthrough: ipv4, [-t filter -C FORWARD -i br-b0f784c11c0d ! -o br-b0f784c11c0d -j ACCEPT]"