rancher: Timeout getting IP address in fresh Ubuntu 16.04 host on 1.3.0
I am aware there are plenty of issues regarding this but since this appears for me in Rancher 1.3 and I’ve tried everything I’m opening this issue nonetheless
Rancher Versions: Server: 1.3 everything else is latest
Docker Version: Docker version 1.12.6 OS and where are the hosts located? (cloud, bare metal, etc): bare metal Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) single node rancher, internal DB Environment Type: (Cattle/Kubernetes/Swarm/Mesos) Cattle
Backstory: We’ve upgraded 5 hosts from Rancher 1.1.3 to 1.3. After 3 days of tinkering most things are running again but two hosts (CentOS 7) always had the infamous “Timeout getting IP address” error so I reinstalled a fresh Ubuntu 16.04 on them.
I added the following to /etc/docker/daemon.json
{
"dns": ["8.8.8.8", "8.8.4.4"]
}
Then restarted systemctl daemon-reload & service docker restart but still ipsec, healthcheck & ipsec-cni-driver get timeouts.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 10
- Comments: 24 (2 by maintainers)
I’m getting this with a 16.04 host as well in Rancher 1.3.3
UPDATE: I was able to solve my issue which stemmed from having a catchall for my real domain. e.g. anything.domain.com would resolve to a specific host. This was causing all kinds of trouble for the rancher internal dns. rancher-metadata was resolving to that system instead of where it was supposed to.
On the affected hosts I set the dns-search to something fictional- “example.org” via the /etc/docker/daemon.json.
{ "dns": ["8.8.8.8", "8.8.4.4"], "dns-search": ["example.org"] }Changind DNS does not work for me. I’m using Rancher 1.4.0.
Just created a new environment (cattle) and added a host (CentOS Atomic), the default services die with this error.
Thank you guys for the solution for this problem on my “netcup” root-server.
penguinxr2 wrote :
On the affected hosts I set the dns-search to something fictional- “example.org” via the /etc/docker/daemon.json. { “dns”: [“8.8.8.8”, “8.8.4.4”], “dns-search”: [“example.org”] }
and after that i did :
systemctl daemon-reload & service docker restart
i tried several solutions, but only this one was the solution
@dj-hedgehog Took a while to get there, I tried pretty much every suggestion I could find related to this issue, which ultimately seemed to be exclusively DNS related. So when I found out it was trying to fetch rancher metadata from the webserver that was running on my domain catchall it gave an SSL certificate error. From that, I figured the DNS pecking order must have rancher’s internal stuff dead last, so setting the search domain to something bogus made those queries fail and rancher would resolve against its internal DNS service. This is my understanding at least.