moby: Swarm mode does not listen on published ports

Description

I have a swarm cluster composed of 3 managers running on AWS. When I create a new service using a docker compose file none of the managers listen on the published ports for the service.

Steps to reproduce the issue:

  1. Create a swarm cluster with 3 managers
  2. Run docker deploy --compose-file docker-compose.yml traefik from one of the managers
  3. Run curl -v localhost:8080 from one of the manager nodes

Describe the results you received:

* Rebuilt URL to: localhost:8080/
*   Trying 127.0.0.1...
* connect to 127.0.0.1 port 8080 failed: Connection refused
* Failed to connect to localhost port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8080: Connection refused

Describe the results you expected:

Should be able to contact the running service

Additional information you deem important (e.g. issue happens only occasionally):

The compose file for creating the service is:

version: "3"
networks:
    base:
      driver: overlay
services:
    traefik:
      image: traefik:1.2.0
      command: -c /dev/null --web --docker --docker.swarmmode --docker.watch --docker.domain=traefik --logLevel=DEBUG
      networks:
        - base
      ports:
        - "80:80"
        - "8080:8080"
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
      deploy:
        placement:
          constraints: [node.role == manager]

Output of docker service inspect --pretty traefik:

ID:		xjqwrdjjnwf1ssovc9ehneis9
Name:		control_traefik
Labels:
 com.docker.stack.namespace=traefik
Service Mode:	Replicated
 Replicas:	1
Placement:Contraints:	[node.role == manager]
ContainerSpec:
 Image:		traefik:1.2.0@sha256:d9d82c52bb091466b167ea1c0f2a27c0032baef786ead275d3c40fb9e4759aaa
 Args:		-c /dev/null --web --docker --docker.swarmmode --docker.watch --docker.domain=traefik --logLevel=DEBUG 
Mounts:
  Target = /var/run/docker.sock
   Source = /var/run/docker.sock
   ReadOnly = false
   Type = bind
Resources:
Networks: v6v9yr3847770cp5hjez9cb60 
Endpoint Mode:	vip
Ports:
 PublishedPort 80
  Protocol = tcp
  TargetPort = 80
 PublishedPort 8080
  Protocol = tcp
  TargetPort = 8080 

Output of sudo netstat -tunap | grep LISTEN:

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1328/sshd      
tcp6       0      0 :::2377                 :::*                    LISTEN      10141/dockerd   
tcp6       0      0 :::4243                 :::*                    LISTEN      10141/dockerd   
tcp6       0      0 :::22                   :::*                    LISTEN      1328/sshd  

(here I was expecting docker to listen to ports 80 and 8080)

Output of sudo iptables -nvL -t nat:

Chain PREROUTING (policy ACCEPT 1186 packets, 71170 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1180 70804 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 1180 packets, 70804 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 542 packets, 37953 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 542 packets, 37953 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    6   366 MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker_gwbridge *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0 

Output of docker version:

Docker version 17.03.0-ce, build 3a232c8

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: 2ezx7ap7kgiv3r6mq5zdoso79
 Is Manager: true
 ClusterID: y5p5sqpgew4tfqy2sr2na4ang
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.31.7.109
 Manager Addresses:
  0.0.0.0:2377
  172.31.21.219:2377
  172.31.7.109:2377
  172.31.8.25:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 486.7 MiB
Name: ip-172-31-7-109
ID: HUMZ:FHWJ:XFYZ:ECNW:4Z5P:7D7L:RW45:7OSL:DPXL:E47P:TUE5:JAVH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 41
 Goroutines: 102
 System Time: 2017-03-26T07:01:09.469308353Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

The docker nodes are running in AWS. The security group attached to the EC2 instances allow all traffic over all protocol between the docker nodes.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 1
  • Comments: 16 (4 by maintainers)

Most upvoted comments

I solved it using an earlier version of “boot2docker”. Apparently version 18.09 has problems.

docker-machine create -d hyperv --hyperv-virtual-switch “myswitch” --hyperv-boot2docker-url=https://github.com/boot2docker/boot2docker/releases/down load/v18.05.0-ce/boot2docker.iso myvm1

I have tested this solution with virtualbox driver (in mac) and with hyper-v (obviously windows)

I have a similar problem. Docker in Swarm mode doesn’t listen any exposed ports when deploying services to Scaleway C2S server. I’ve investigated docker logs with journalctl -u docker.service and found some errors like Apr 11 19:40:18 scw-a728ca dockerd[12709]: time="2017-04-11T19:40:18Z" level=error msg="setting up rule failed, [-t mangle -D OUTPUT -d 10.0.0.8/32 -j MARK --set-mark 323]: (iptables failed: iptables --wait -t mangle -D OUTPUT -d 10.0.0.8/32 -j MARK --set-mark 323: iptables: No chain/target/match by that name.\n (exit status 1))" Apr 11 19:40:18 scw-a728ca dockerd[12709]: time="2017-04-11T19:40:18.259536582Z" level=error msg="Failed to delete firewall mark rule in sbox 097582e (c8f11d8): reexec failed: exit status 5" Apr 11 19:40:18 scw-a728ca dockerd[12709]: time="2017-04-11T19:40:18Z" level=info msg="Firewalld running: false" Apr 11 19:40:18 scw-a728ca dockerd[12709]: time="2017-04-11T19:40:18Z" level=error msg="setting up rule failed, [-t mangle -D PREROUTING -p tcp --dport 8000 -j MARK --set-mark 324]: (iptables failed: iptables --wait -t mangle -D PREROUTING -p tcp --dport 8000 -j MARK --set-mark 324: iptables: No chain/target/match by that name.\n (exit status 1))" Apr 11 19:40:18 scw-a728ca dockerd[12709]: time="2017-04-11T19:40:18.478598359Z" level=error msg="Failed to delete firewall mark rule in sbox ingress (ingress): reexec failed: exit status 5" Apr 11 19:40:18 scw-a728ca dockerd[12709]: time="2017-04-11T19:40:18Z" level=info msg="Firewalld running: false"

However everything with exactly the same configuration works perfectly on DigitalOcean hosted servers. No any errors are logged.