moby: docker swarm mode overlay can't access external internet

@fingermark Your observation that using --subnet=192.168.1.0/24 for overlay fixes the problem is correct. The reason for the original problem is a bit involved though.

Within a container in an EC2 instance on a private subnet only the name resolution fails, but the external connectivity is just fine…

root@6ce46f8458c1:/# ping google.com
^C
root@6ce46f8458c1:/# ping 216.58.193.110
PING 216.58.193.110 (216.58.193.110) 56(84) bytes of data.
64 bytes from 216.58.193.110: icmp_seq=1 ttl=45 time=7.60 ms
64 bytes from 216.58.193.110: icmp_seq=2 ttl=45 time=7.25 ms
^C
--- 216.58.193.110 ping statistics ---

The address range in VPC by default is (10.0.0.0/16) and the DHCP server provisioned by AWS is 10.0.0.2. The default address range of overlay networks also start at 10.0.0.0/24. This is still fine because the docker overlay bridge is created inside a separate network namespace (this has a kernel version dependency though). But for DNS resolution, the queries from the container are sent to the Docker embedded DNS server, 127.0.0.11 (address in container’s resolv.conf).

Though the DNS server runs as part of the docker daemon it tries to do the routing to the DNS name server address, 10.0.0.2 here, inside the container’s name space. Since it conflicts with the overlay address range, packets will not be sent to the DNS server. That is the reason why external connectivity works but only the resolution is failing.

To work around this there are two options…

1. creating the overlay network in a different address range like you did
2. In the EC2 host, change the DNS server IP in the resolv.conf. for ex: if you change it to 8.8.8.8 it will work.

Since there is no fix required here I think this issue can be closed.

@herbrandson I wound up starting over. I left the docker_gwbridge gateway alone and assigned my network overlay a subnet different from anything else: --subnet=192.168.1.0/24. I’m now able to access servers on the same network and externally.

Maybe it would help if I described exactly what I was looking to achieve:

Using AWS EC2 t2.micro instances and AWS VPC with a private and public subnet. How would I create the following using docker swarm mode:

Requirements

Hub Node (count: 1)

• On a publicly accessible IP
• Must be on same public IP every time
• Must be able to communicate with chrome nodes
• Run with: docker service create --name hub --network seleniumnet -p 4444:4444 selenium/hub --replicas 1

Chrome Nodes (count: 50)

• On private IPs (without public IP)
• Must be able to make requests to outside internet
• Must be able to make requests to Hub Node
• Only one node per host
• Run with: docker service create --network seleniumnet --endpoint-mode dnsrr --name chrome --mount type=bind,source=/dev/shm,target=/dev/shm -e HUB_PORT_4444_TCP_ADDR=hub -e HUB_PORT_4444_TCP_PORT=4444 --replicas 50 selenium/node-chrome:2.53.1-beryllium bash -c 'REMOTE_HOST=http://$HOSTNAME:5555 /opt/bin/entry_point.sh'

AWS VPC Details:

VPC (vpc-00000001):

• subnet: 10.0.0.0/16

public subnet (subnet-00000001):

• subnet: 10.0.0.0/24
• routes:

10.0.0.0/16 local
0.0.0.0/0 igw-id

private subnet (subnet-00000002):

• subnet: 10.0.1.0/24
• routes:

10.0.0.0/16 local
0.0.0.0/0 nat-gateway-id

Notes:

docker_gwbridge gets automatically created with:

"Config": [
    {
        "Subnet": "172.18.0.0/16",
        "Gateway": "172.18.0.1"
    }

It would be great if this could be stated in docker swarm create network documentation. I spent several hours figuring this out …

@fingermark Per your comments above, I ran the following commands

docker network disconnect -f docker_gwbridge gateway_ingress-sbo
docker network rm docker_gwbridge
docker network create --opt com.docker.network.bridge.enable_icc=false --subnet=10.0.0.0/16 docker_gwbridge

However I’m still unable to launch services due to failed to allocate gateway (1…

Did I miss a step?