swarmkit: Service is not DNS resolvable from another one if containers run on different nodes

I got it working now. Here are some insights that may help others:

• Don’t try to use docker for Windows to get multi-node mesh network (swarm) running. It’s simply not (yet) supported. If you google around, you find some Microsoft blogs telling about it. Also the docker documentation mentions it somewhere. It would be nice, if docker cmd itself would print an error/warning when trying to set something up under Windows - which simply doesn’t work. It does work on a single node though.
• Don’t try to use a Linux in a Virtualbox under Windows and hoping to workaround with it. It, of course, doesn’t work since it has the same limitations as the underlying Windows.
• Make sure you open ports at least 7946 tcp/udp and 4789 udp for worker nodes. For master also 2377 tcp. Use e.g. netcat -vz -u <ip> <port> for udp check. Without -u for tcp.
• Make sure to pass --advertise-addr on the docker worker node (!) when executing the join swarm command. Here put the external IP address of the worker node which has the mentioned ports open. Doublecheck that the ports are really open!
• Using ping to check the DNS resolution for container names works. If you forget the --advertise-addr or not opening port 7946 results in DNS resolution not working on worker nodes!

My main fault was using Windows and not specifying --advertise-addr - since I thought the IP address of the master was already specified correctly by the generated join token cmd. But it’s important to specify the worker node IP as well on join!

I hope that helps someone. Most of the stuff is mentioned in the documentation and here in the comments. But only the combination of the mentioned points worked for me.

BTW: I’ve tested this with docker-compose v3.3 syntax and deployed it via docker stack deploy with the default overlay network. As a kernel I used the Ubuntu 16.04 LTS, 4.4.0-93-generic kernel.

So I’m still stumped by this…

Given three AWS nodes running in a private VPC subnet with the security group set to allow all traffic in and out on all ports, both UDP, and TCP on the subnet 11.0.0.0/8, I still cannot get obtain the IP address of services running on other nodes in the docker swarm. Any services running on a node in the swarm can get IP address for services running on the same node.

How to reproduce… 1 - Create an attachable network (docker-compose version 3 files still don’t support attachable overlay networks)

docker network create --driver overlay --attachable --subnet 192.168.1.0/24 alpha_net

2 - Start the following docker-compose file with docker stack deploy --compose-file=docker-compose.yml alpha This is a stripped down sample that creates a consul cluster. I’ve left out some of the other services from the compose file.

version: "3.1"

services:

  # Consul server 1 of 3
  consul1:
    image: consul:0.7.5
    command: agent -bind=0.0.0.0 -client=0.0.0.0 -advertise='{{ GetAllInterfaces | include "network" "192.168.1.0/24" | attr "address" }}' -log-level=INFO -node=config-server-1 -server -bootstrap-expect=3 -rejoin -retry-join=consul1 -retry-join=consul2 -retry-join=consul3
    environment:
      SERVICE_8500_IGNORE: "true"
      SERVICE_8300_IGNORE: "true"
      SERVICE_8301_IGNORE: "true"
      SERVICE_8302_IGNORE: "true"
      SERVICE_8400_IGNORE: "true"
      SERVICE_8600_IGNORE: "true"
    networks:
      - alpha_net
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - 'node.labels.cpu == enabled'

  # Consul server 2 of 3
  consul2:
    image: consul:0.7.5
    command: agent -bind=0.0.0.0 -client=0.0.0.0 -advertise='{{ GetAllInterfaces | include "network" "192.168.1.0/24" | attr "address" }}' -log-level=INFO -node=config-server-2 -server -bootstrap-expect=3 -rejoin -retry-join=consul1 -retry-join=consul2 -retry-join=consul3
    environment:
      SERVICE_8500_IGNORE: "true"
      SERVICE_8300_IGNORE: "true"
      SERVICE_8301_IGNORE: "true"
      SERVICE_8302_IGNORE: "true"
      SERVICE_8400_IGNORE: "true"
      SERVICE_8600_IGNORE: "true"
    networks:
      - alpha_net
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - 'node.labels.cpu == enabled'

  # Consul server 3 of 3
  consul3:
    image: consul:0.7.5
    command: agent -bind=0.0.0.0 -client=0.0.0.0 -advertise='{{ GetAllInterfaces | include "network" "192.168.1.0/24" | attr "address" }}' -log-level=INFO -node=config-server-3 -server -bootstrap-expect=3 -rejoin -retry-join=consul1 -retry-join=consul2 -retry-join=consul3
    environment:
      SERVICE_8500_IGNORE: "true"
      SERVICE_8300_IGNORE: "true"
      SERVICE_8301_IGNORE: "true"
      SERVICE_8302_IGNORE: "true"
      SERVICE_8400_IGNORE: "true"
      SERVICE_8600_IGNORE: "true"
    networks:
      - alpha_net
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - 'node.labels.cpu == enabled'
networks:
  alpha_net:
    external: true

The above fails, since container running consul1 cannot resolve consul2 and consul3 into IP addresses.

    2017/02/22 01:58:58 [INFO] agent: (LAN) joining: [consul1 consul2 consul3]
    2017/02/22 01:58:58 [WARN] memberlist: Failed to resolve consul2: lookup consul2 on 127.0.0.11:53: no such host
    2017/02/22 01:58:58 [WARN] memberlist: Failed to resolve consul3: lookup consul3 on 127.0.0.11:53: no such host
    2017/02/22 01:58:58 [INFO] agent: (LAN) joined: 1 Err: <nil>
    2017/02/22 01:58:58 [INFO] agent: Join completed. Synced with 1 initial agents
    2017/02/22 01:59:05 [ERR] agent: failed to sync remote state: No cluster leader

And, if I manually attach to the container for consul1 with docker exec -it <container_id> /bin/sh, I can nslookup services running on the same node, but not services running on a different node.

/ # nslookup consul1
Name:      consul1
Address 1: 192.168.1.31 ip-192-168-1-31.us-west-2.compute.internal
/ # nslookup consul2
nslookup: can't resolve 'consul2': Name does not resolve
/ # nslookup consul3
nslookup: can't resolve 'consul3': Name does not resolve
/ # nslookup docdb
nslookup: can't resolve 'docdb': Name does not resolve
/ # nslookup userdb
Name:      userdb
Address 1: 192.168.1.37 ip-192-168-1-37.us-west-2.compute.internal

(userdb in the list above is another service in the compose file… left out for brevity’s sake)

I can reach the name server at 127.0.0.11 just fine inside the container for consul1, but it seems as if IP addresses for services running on other nodes aren’t getting synchronized in the swarm network.

Do you have TCP port 7946 open on your hosts? Gossip needs that port open for networking to work correctly.

In my case, it turned out to be related to ports. It worked after following ports were accessible between nodes:

• TCP port 2377 for cluster management communications
• TCP and UDP port 7946 for communication among nodes
• UDP port 4789 for overlay network traffic

Reference: https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts

Looking at the same issue in my setup, but I’m noticing something odd. I have 1 manager and 8 worker nodes. 5 of my 8 worker nodes fail to resolve the service name over the overlay network. The other 3 have no issue in doing so. No matter what service I launch or how I launch it, so long as it’s connected to the correct overlay network, the same 3 have no issue resolving by service name.

I have absolutely no idea why the other 5 nodes in my swarm continue to have problems. I’ve tried the quick fixes listed in this thread to no avail. Each of my worker nodes are identical copies of each other.

--advertise-addr was the silver bullet for us. Documentation says you can use this switch with NIC’s name like --advertise-addr eth0:2377 where eth0 is address independent and fits your requirement of nodes with dynamic IP address. Same as we have.

See –advertise-addr value

I got it working now. Here are some insights that may help others:

• Don’t try to use docker for Windows to get multi-node mesh network (swarm) running. It’s simply not (yet) supported. If you google around, you find some Microsoft blogs telling about it. Also the docker documentation mentions it somewhere. It would be nice, if docker cmd itself would print an error/warning when trying to set something up under Windows - which simply doesn’t work. It does work on a single node though.
• Don’t try to use a Linux in a Virtualbox under Windows and hoping to workaround with it. It, of course, doesn’t work since it has the same limitations as the underlying Windows.
• Make sure you open ports at least 7946 tcp/udp and 4789 udp for worker nodes. For master also 2377 tcp. Use e.g. netcat -vz -u for udp check. Without -u for tcp.
• Make sure to pass --advertise-addr on the docker worker node (!) when executing the join swarm command. Here put the external IP address of the worker node which has the mentioned ports open. Doublecheck that the ports are really open!
• Using ping to check the DNS resolution for container names works. If you forget the --advertise-addr or not opening port 7946 results in DNS resolution not working on worker nodes!

My main fault was using Windows and not specifying --advertise-addr - since I thought the IP address of the master was already specified correctly by the generated join token cmd. But it’s important to specify the worker node IP as well on join!

I hope that helps someone. Most of the stuff is mentioned in the documentation and here in the comments. But only the combination of the mentioned points worked for me.

BTW: I’ve tested this with docker-compose v3.3 syntax and deployed it via docker stack deploy with the default overlay network. As a kernel I used the Ubuntu 16.04 LTS, 4.4.0-93-generic kernel.

Almost googled my ass off to finally find this valuable suggestion. This windows platform issue really bugs me.

I’m seeing this too. I’m using Docker for AWS and this has happened both on beta4 and now on beta5. Service names are sometimes unresolvable, sometimes resolvable but no route to host. It also works sometimes. I’ve been so far unable to reliably reproduce from scratch.