kubernetes: Unable to resolve hostname using `kubectl logs`

Using kubectl logs shows the following error:

Error from server: Get https://node-1:10250/containerLogs/kube-system/kibana-logging-v1-x1hr3/kibana-logging: dial tcp: lookup node-1: no such host

Kubernetes master uses nodeName to fetch logs from kubelet, which doesn’t always resolve. There are two workarounds as I can think of:

Manually add host entry in master’s /etc/hosts, but this doesn’t scale;
Use hostname override to use raw IP address. This is better, but raw IP is hard to manage and we see this error https://github.com/kubernetes/kubernetes/issues/22109

Since master already have the mapping from nodeName to IPAddress, we can get rid of the problem by just using internal IP?

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 14
Comments: 24 (9 by maintainers)

Commits related to this issue

Use nodeutil.GetHostIP consistently when talking to nodes Most of our communications from apiserver -> nodes used nodutil.GetNodeHostIP, but a few places didn't - and this meant that the node name ne... — committed to justinsb/kubernetes by justinsb 8 years ago
Hostname alias fixes Change the kubelet --hostname-override flag to use the ansible_hostname variable which should be more consistent with the value required by cloud providers Add ansible_hostname ... — committed to chadswen/kargo by chadswen 8 years ago
couldn't fetch log from dashboard or run kubectl exec: kubernetes/kubernetes#22770 — committed to fouadsemaan/ansible-kubernetes-cluster by fouadsemaan 7 years ago
kubernetes/kubernetes#22770 couldn't fetch log from dashboard or run kubectl exec (#8) * couldn't fetch log from dashboard or run kubectl exec: kubernetes/kubernetes#22770 * put kubelet preferred... — committed to andrewrothstein/ansible-kubernetes-cluster by fouadsemaan 7 years ago

Most upvoted comments

This is about to become a much bigger problem as it affects everyone using the new kubeadm tool: https://lukemarsden.github.io/docs/getting-started-guides/kubeadm/#limitations

+21

lukemarsden on Sep 26, 2016

I confirm that kubectl logs and kubectl exec work when using Kubernetes 1.5.0-beta3 and having set --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP for apiserver.

+14

xmik on Dec 10, 2016

We should be using the nodeutil.GetNodeHostIP, not the NodeName. It is a long standing bug. I’ll try to put together a PR.

justinsb on Sep 29, 2016

I also encountered a similar problem, weave network Kai not.

[root@master schema]# kubectl get po --namespace=kube-system -o wide
NAME                             READY     STATUS              RESTARTS   AGE       IP          NODE
etcd-master                      1/1       Running             3          10h       10.0.2.15   master
kube-apiserver-master            1/1       Running             3          10h       10.0.2.15   master
kube-controller-manager-master   1/1       Running             3          10h       10.0.2.15   master
kube-discovery-982812725-mtqfh   1/1       Running             0          18m       10.0.2.15   master
kube-dns-2247936740-3k2gn        0/3       ContainerCreating   0          5m        <none>      node1
kube-proxy-amd64-izxxe           1/1       Running             3          10h       10.0.2.15   master
kube-proxy-amd64-z8nj8           1/1       Running             1          10h       10.0.2.15   node1
kube-scheduler-master            1/1       Running             3          10h       10.0.2.15   master
weave-net-7m5jm                  2/2       Running             8          10h       10.0.2.15   master
weave-net-ed15l                  1/2       CrashLoopBackOff    1          20s       10.0.2.15   node1

[root@node1 schema]# kubectl logs -f weave-net-ed15l --namespace=kube-system -c weave
Error from server: Get https://node1:10250/containerLogs/kube-system/weave-net-ed15l/weave?follow=true: dial tcp: lookup node1 on 10.0.2.3:53: no such host

markthink on Oct 13, 2016

@mkulke minor correction (because I just ran into it), use --kubelet-preferred-address-types.

bonovoxly on Dec 1, 2016

I haven’t had the time to test an 1.5 cluster yet, but from looking at the code, i guess what works now is:

Set something master-resolvable via --hostname-override on the kubelet.
Have the cloud-provider rename the node to something non-resolvable.

@hectorj2f By default the node uses the node’s hostname, so when you use --hostname-override with some non-resolvable display name (e.g. “my-node-1”), the master will not be able to talk to the kubelet, because by default the hostname will be preferred as means to talk to the kubelet.

however in 1.5 there is also --kubelet-preferred-address-types (https://github.com/kubernetes/kubernetes/blob/release-1.5/cmd/kube-apiserver/app/options/options.go#L112), which you can use to specify a priority list of comm options (Hostname, InternalIP, ExternalIP, LegacyHostIP).

The flag is not documented yet, but I think this should solve most issues around apiserver->kubelet comm.

mkulke on Dec 1, 2016

We also have a similar problem. We defined a specific hostname (not resolvable) to get a customized node name. We also specify the node-ip to a resolvable IP address in the kubelet parameters. So we expected the master api to use that node-ip instead of the hostname or node name.

Surprisingly for us, kubernetes master uses the hostname to communicate with the kubelet instead of the node ip. So it cannot fetch the logs of our apps.

hectorj2f on Nov 21, 2016