rancher: Improve error logs in health check containers when health check is paused when ipsec is not in "active" state.

I’ve faced similar issues.

Solution:

• Review the address on Rancher UI -> Hosts, check if the ip of the hosts correct or not.
• Add Host, filling Step 4 with the docker server’s IP
• run the docker agent with Step 5 command.

In my case, our environment had been destroied in some reason, then we are going to recreate all the compoent. I’ve tried to start the agent without specified agent located server’s IP, when the second hosts set up, the issue comes up. Then I check the logs of the agent, found that DETECTED_CATTLE_AGENT_IP is wrong, and the IP of the host on the Rancher UI shows the same one for each hosts, and it might lead to the rancher trying to edit the same record in the DB, that’ll caused the dead lock.

P.S. If you’re facing the same issue, you could specified the IP for fixing in time, then contact to your network administrator to troubleshoot the network settings.

Not quite sure if another repro matters, but I can easily reproduce the issue using one physical (ubuntu 17.10) machine running v1.6.15 on the host and two docker-machine vm nodes. The following code creates kvm vms, but virtualbox works equally.

docker-machine create \
    --driver kvm \
    --kvm-cpu-count 2 \
    --kvm-memory 1092 \
    --kvm-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v17.12.1-ce/boot2docker.iso \
    kvm-1712-1
    
docker-machine create \
    --driver kvm \
    --kvm-cpu-count 2 \
    --kvm-memory 1092 \
    --kvm-boot2docker-url https://github.com/boot2docker/boot2docker/releases/download/v17.12.1-ce/boot2docker.iso \
    kvm-1712-2    

Start from an empty environment and create the first custom host, “docker-machine ssh kvm-1712-1” and paste the command line from rancher.

The services go up and everything is green.

Now add the second node.

ipsec/healthcheck go Up/Down/Initializing/Unhealthy

ipsec-router logs show entries such as:

3/22/2018 3:49:51 PMtime="2018-03-22T14:49:51Z" level=info msg="samonitor: expected SA for host: 172.17.42.1, but not found."
3/22/2018 3:49:51 PMtime="2018-03-22T14:49:51Z" level=error msg="samonitor: error initiating missing SA child-172.17.42.1: unsuccessful Initiate: CHILD_SA config 'child-172.17.42.1' not found"

healthcheck logs go:

3/22/2018 3:55:35 PMtime="2018-03-22T14:55:35Z" level=error msg="Failed to report status 8734367f-1c29-4d3b-8214-e939fc06f058_5a30ec2b-3a98-481b-be9e-11138b7fbff0_1=DOWN: Bad response from [http://.../v1/serviceevents], statusCode [409]. Status [409 Conflict]. Body: [{\"id\":\"16d637fd-f0eb-487d-b94a-2c6511597b4b\",\"type\":\"error\",\"links\":{},\"actions\":{},\"status\":409,\"code\":\"Conflict\",\"message\":\"Conflict\",\"detail\":null,\"baseType\":\"error\"}]"

Updated system packages (apt, since I’m on Ubuntu) and updated to:

• Docker 17.12.0-ce
• rancher-agent v1.2.8
• rancher-server v1.6.14

still have the problem.

I can afford to reboot the machine and restart docker, but unfortunately I can’t afford more than 3 nodes.

I’m using DHCP and ran into this issue as well. (Though my DHCP address has not changed)

Restarting the network stack in ROS resolved the issue I was having with the health check service.

fwiw I have the same setup as @mjaverto : server is a DNS entry that resolves to private IP; hosts are all private IPs in same subnet. have rebuilt hosts from bare VM several times and re-registered to no avail; server also. >1 host will reproduce the issue, but exactly one host will run fine (doesn’t matter which).