rancher: Rancher agent not connecting to Rancher server on same host with "Custom" Add Host

I am running Rancher server in the provided container on DigitalOcean (exposing 8080) and the admin seems to work fine. My firewall rules seems reasonable:

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
2376/tcp                   ALLOW       Anywhere
3376/tcp                   ALLOW       Anywhere
500/udp                    ALLOW       Anywhere
4500/udp                   ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
2376/tcp (v6)              ALLOW       Anywhere (v6)
3376/tcp (v6)              ALLOW       Anywhere (v6)
500/udp (v6)               ALLOW       Anywhere (v6)
4500/udp (v6)              ALLOW       Anywhere (v6)

But when I want to add the same host, as an agent, it hangs on attempting to connect to the master.

I click through the admin interface for “custom”, and it tells me to do:

$ docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.7.11 http://<ip>:8080/v1/scripts/328B791E15E99D370F1D:1438639200000:8aQ7dtSOvfPpOLb1OzuWIMsOKoQ

When I run that locally, I get logs in the container like:

INFO: Running Agent Registration Process, CATTLE_URL=<ip>:8080/v1
INFO: Checking for Docker version >= 1.6.0
INFO: Found Server version: 1.7.1
INFO: docker version: Client version: 1.6.0
INFO: docker version: Client API version: 1.18
INFO: docker version: Go version (client): go1.4.2
INFO: docker version: Git commit (client): 4749651
INFO: docker version: OS/Arch (client): linux/amd64
INFO: docker version: Server version: 1.7.1
INFO: docker version: Server API version: 1.19
INFO: docker version: Go version (server): go1.4.2
INFO: docker version: Git commit (server): 786b29d
INFO: docker version: OS/Arch (server): linux/amd64
INFO: docker info: Containers: 6
INFO: docker info: Images: 46
INFO: docker info: Storage Driver: aufs
INFO: docker info: Root Dir: /var/lib/docker/aufs
INFO: docker info: Backing Filesystem: extfs
INFO: docker info: Dirs: 58
INFO: docker info: Dirperm1 Supported: false
INFO: docker info: Execution Driver: native-0.2
INFO: docker info: Kernel Version: 3.13.0-57-generic
INFO: docker info: Operating System: Ubuntu 14.04.2 LTS
INFO: docker info: CPUs: 2
INFO: docker info: Total Memory: 1.955 GiB
INFO: docker info: Name: beefydrop
INFO: docker info: ID: AG5H:ECGI:PIYD:HJO4:S7FY:WBPC:YWSB:YJ2H:FGO2:6A5I:7TGR:IHXE
INFO: docker info: Http Proxy:
INFO: docker info: Https Proxy:
INFO: docker info: No Proxy:
WARNING: No swap limit support
INFO: docker info: Labels:
INFO: docker info: provider=digitalocean
INFO: Attempting to connect to: http://<ip>:8080/v1
WARNING: No swap limit support
ERROR: http://<ip>:8080/v1 is not accessible

I managed to add another host fine. What’s going on? Other issues (https://github.com/rancher/rancher/issues/872) suggested -e CATTLE_AGENT_IP=<ip>, but that didn’t work for me.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 20

Most upvoted comments

I’ve fixed! The problem is with my certificate. In the server logs, these messages where being displayed every second:

time="2016-01-06T11:24:23Z" level=fatal msg="Unable to create event router" error="Get https://ranchermgmt.socialbase.com.br/v1: x509: certificate signed by unknown authority" 
time="2016-01-06T11:24:24Z" level=info msg="Setting log level" logLevel=info 

After correcting the certificate issue, restarting the server, go to UI, adding a host again and re-run the agent with this command, the issue was fixed:

docker run -e CATTLE_AGENT_IP=172.17.0.2 -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 https://ranchermgmt.socialbase.com.br/v1/scripts/11A1DA6CF21B26A360AF:1452078000000:QD6RuI30vvfrzRiMlvJk0T4k

Please reopen this case.

I’ve created a new EC2 instance using this ami ID on AWS EC2: rancheros-v0.4.2-hvm-0 (ami-53045239). After the instance is reachable through ssh, I’ve setup the rancher server container on it using this command:

[root@ip-172-30-2-89 ~]# docker run -d --name=rancher-server --restart=always -p 8080:8080 rancher/server 

After the rancher server’s container is running, I’ve installed a nginx container for proxy requests on HTTPs/443 to 8080 of rancher server’s container:

mkdir -p /opt/nginx/ssl

cat <<EOF> /opt/nginx/rancher.conf
upstream rancher {
    server rancher-server:8080;
}

server {
    listen 443 ssl;
    server_name ranchermgmt.mydomain.com;
    ssl_certificate /etc/nginx/ssl/ranchermgmt.mydomain.com.crt;
    ssl_certificate_key /etc/nginx/ssl/ranchermgmt.mydomain.com.key;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

server {
    listen 80;
    server_name ranchermgmt.mydomain.com;
    return 301 https://$server_name$request_uri;
}

EOF

docker run -d -p 80:80 -p 443:443 -v /opt/nginx/ssl:/etc/nginx/ssl/:ro -v /opt/nginx/rancher.conf:/etc/nginx/conf.d/rancher.conf:ro --link='rancher-server' nginx

With these two containers running, I was able to connect to https://ranchermgmt.mydomain.com using SSL/443 with my own certificate using any browser.

Very well, now I want to add this EC2 instance as a host to the rancher server container that is running inside it. For doing it, I’ve first enabled a local authentication on rancher server, by using it’s web ui. After that, I’ve did the procedure described here: http://docs.rancher.com/rancher/rancher-ui/infrastructure/hosts/custom/#adding-hosts-to-the-same-machine-as-rancher-server, but the procedure not works.

These are my host interfaces:

[root@ip-172-30-2-89 ~]# ifconfig -a
docker-sys Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:172.18.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::447e:a9ff:feef:f906/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:368 (368.0 B)  TX bytes:648 (648.0 B)

docker0   Link encap:Ethernet  HWaddr 02:42:D5:37:76:1F  
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:d5ff:fe37:761f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:178439 errors:0 dropped:0 overruns:0 frame:0
          TX packets:176795 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:36347151 (34.6 MiB)  TX bytes:45152492 (43.0 MiB)

eth0      Link encap:Ethernet  HWaddr 0E:C3:1F:3F:3B:6F  
          inet addr:172.30.2.89  Bcast:172.30.2.255  Mask:255.255.255.0
          inet6 addr: fe80::cc3:1fff:fe3f:3b6f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:577172 errors:0 dropped:0 overruns:0 frame:0
          TX packets:216445 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:599497248 (571.7 MiB)  TX bytes:43973702 (41.9 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

none      Link encap:Ethernet  HWaddr BA:AF:8D:2C:1A:6C  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

veth2e82b72 Link encap:Ethernet  HWaddr 46:14:59:6E:D2:C6  
          inet6 addr: fe80::4414:59ff:fe6e:d2c6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:110996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77282 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12322539 (11.7 MiB)  TX bytes:30958505 (29.5 MiB)

veth71a4990 Link encap:Ethernet  HWaddr BA:0B:9D:DB:DD:16  
          inet6 addr: fe80::b80b:9dff:fedb:dd16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:57864 errors:0 dropped:0 overruns:0 frame:0
          TX packets:95689 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:27660318 (26.3 MiB)  TX bytes:11314916 (10.7 MiB)

These are the security groups that my EC2 instance is associated:

image

I’ve tried to create many containers with many CATTLE_AGENT_IPs, but all of them reports this error:

INFO: Attempting to connect to: https://ranchermgmt.mydomain.com/v1
ERROR: https://ranchermgmt.mydomain.com/v1 is not accessible

These are the commands that I’ve executed to create the containers:

docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 https://ranchermgmt.mydomain.com/v1/scripts/11A1DA6CF21B26A360AF:1452027600000:KoczyizHYyqPLeetTDhFcJk5pn0 -e CATTLE_AGENT_IP=172.17.0.3 

docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 https://ranchermgmt.mydomain.com/v1/scripts/11A1DA6CF21B26A360AF:1452027600000:KoczyizHYyqPLeetTDhFcJk5pn0 -e CATTLE_AGENT_IP=172.30.2.89 

docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 https://ranchermgmt.mydomain.com/v1/scripts/11A1DA6CF21B26A360AF:1452027600000:KoczyizHYyqPLeetTDhFcJk5pn0 -e CATTLE_AGENT_IP=172.17.0.1

docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock rancher/agent:v0.8.2 https://ranchermgmt.mydomain.com/v1/scripts/11A1DA6CF21B26A360AF:1452027600000:KoczyizHYyqPLeetTDhFcJk5pn0 -e CATTLE_AGENT_IP=172.18.42.1

Even using the latest rancher agent image, the agent not works

I can’t resolve this issue by using ufw because rancheros 0.4.2 doesn’t have ufw instaled. But, even using iptables directly, I think it isn’t necessary because iptables INPUT Chain in openned by default:

[root@ip-172-30-2-89 ~]# iptables  -L -n -v
Chain INPUT (policy ACCEPT 18960 packets, 170M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 165K   42M DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
82471   35M ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 160K   33M ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
  242 14520 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER     all  --  *      docker-sys  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker-sys  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker-sys !docker-sys  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker-sys docker-sys  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 13561 packets, 1958K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:8080
81865 6789K ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.3           tcp dpt:443
   29  1638 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.3           tcp dpt:80

[root@ip-172-30-2-89 ~]# iptables  -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 29173 packets, 2004K bytes)
 pkts bytes target     prot opt in     out     source               destination         
11311  678K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 8 packets, 424 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 307 packets, 23653 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 11828 packets, 715K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
28964 1992K MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      docker-sys  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
    0     0 MASQUERADE  all  --  *      !docker-sys  172.18.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       172.17.0.3           172.17.0.3           tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       172.17.0.3           172.17.0.3           tcp dpt:80

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    3   180 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:172.17.0.2:8080
11275  677K DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.17.0.3:443
    8   380 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.17.0.3:80

I need a solution for this if possible…

Ok, I reproduced and adding ufw allow 8080/tcp to the firewall rules fixed the problem.

The questionable thing was why were you able to get to the UI/API via the browser. Long story short, the iptables rule that was created when you did docker run ip 8080:8080 ... was allowing external traffic in, but not internal traffic. Internal traffic was being blocked by ufw.