kubernetes: dns can't resolve kubernetes.default and/or cluster.local

/kind bug What happened: I’ve setup Kubernetes cluster on Ubuntu 18.04, v1.11.1:

KubeDNS:

$ kubectl get svc -n kube-system
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   10d
kubernetes-dashboard   ClusterIP   10.99.230.158    <none>        443/TCP         4d
tiller-deploy          ClusterIP   10.111.190.156   <none>        44134/TCP       8d

Version:

$ kubectl version --short
Client Version: v1.11.1
Server Version: v1.11.1

When I run busybox for testing:

kubectl create -f https://k8s.io/examples/admin/dns/busybox.yaml

I am getting this:

$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server:		10.96.0.10
Address:	10.96.0.10:53

** server can't find kubernetes.default: NXDOMAIN

*** Can't find kubernetes.default: No answer

$ kubectl exec -ti busybox -- nslookup cluster.local
Server:		10.96.0.10
Address:	10.96.0.10:53

** server can't find cluster.local: NXDOMAIN

*** Can't find cluster.local: No answer

What you expected to happen:

I expect the kubernetes.default or cluster.local to be resolved.

How to reproduce it (as minimally and precisely as possible): Maybe try to install new k8s cluster on Ubuntu 18.04 following official instructions.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: Bare metal, OVH, Ubuntu 18.04
  • OS (e.g. from /etc/os-release):
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  • Kernel (e.g. uname -a):
$ uname -a
Linux kubernetes-slave 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others: These are my pods:
$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name
pod/coredns-78fcdf6894-c4sk8
pod/coredns-78fcdf6894-mzv9t
pod/kube-dns-569b8c4c96-bwwvm

Here are pod logs:

$ kubectl logs --namespace=kube-system kube-dns-569b8c4c96-bwwvm -c sidecar
ERROR: logging before flag.Parse: W0802 17:51:49.028526       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:59054->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:51:54.029062       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:51343->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:51:59.029389       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:58205->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:04.029922       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:37475->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:09.030484       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:39067->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:14.030962       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:38175->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:19.031436       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:56535->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:24.031820       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:57310->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:29.032374       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:37181->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:34.032952       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:37284->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:39.033511       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:51098->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:44.034022       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:36836->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:49.034444       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:57543->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:54.034865       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:38068->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:52:59.035304       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:59394->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:04.035717       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:36127->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:09.036246       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:42850->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:14.036602       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:43571->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:19.037163       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:45439->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:24.037654       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:35007->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:29.038002       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:46336->127.0.0.1:53: read: connection refused
ERROR: logging before flag.Parse: W0802 17:53:34.038500       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:50540->127.0.0.1:53: read: connection refused

$ kubectl logs --namespace=kube-system kube-dns-569b8c4c96-bwwvm -c dnsmasq
I0802 17:53:35.100942       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0802 17:53:35.101079       1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0802 17:53:35.336808       1 nanny.go:111]
W0802 17:53:35.336832       1 nanny.go:112] Got EOF from stdout
I0802 17:53:35.336849       1 nanny.go:108] dnsmasq[18]: started, version 2.78-security-prerelease cachesize 1000
I0802 17:53:35.336870       1 nanny.go:108] dnsmasq[18]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0802 17:53:35.336877       1 nanny.go:108] dnsmasq[18]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0802 17:53:35.336880       1 nanny.go:108] dnsmasq[18]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0802 17:53:35.336883       1 nanny.go:108] dnsmasq[18]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0802 17:53:35.336887       1 nanny.go:108] dnsmasq[18]: reading /etc/resolv.conf
I0802 17:53:35.336895       1 nanny.go:108] dnsmasq[18]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0802 17:53:35.336901       1 nanny.go:108] dnsmasq[18]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0802 17:53:35.336907       1 nanny.go:108] dnsmasq[18]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0802 17:53:35.336912       1 nanny.go:108] dnsmasq[18]: using nameserver 10.125.211.1#53
I0802 17:53:35.336917       1 nanny.go:108] dnsmasq[18]: using nameserver 10.96.0.10#53
I0802 17:53:35.336922       1 nanny.go:108] dnsmasq[18]: using nameserver 213.186.33.99#53
I0802 17:53:35.336939       1 nanny.go:108] dnsmasq[18]: read /etc/hosts - 7 addresses

$ kubectl logs --namespace=kube-system kube-dns-569b8c4c96-bwwvm -c kubedns
I0802 17:49:38.070785       1 dns.go:48] version: 1.14.4-2-g5584e04
I0802 17:49:38.071345       1 server.go:66] Using configuration read from ConfigMap: kube-system:kube-dns
I0802 17:49:38.071371       1 server.go:113] FLAG: --alsologtostderr="false"
I0802 17:49:38.071379       1 server.go:113] FLAG: --config-dir=""
I0802 17:49:38.071383       1 server.go:113] FLAG: --config-map="kube-dns"
I0802 17:49:38.071387       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0802 17:49:38.071390       1 server.go:113] FLAG: --config-period="10s"
I0802 17:49:38.071394       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0802 17:49:38.071397       1 server.go:113] FLAG: --dns-port="10053"
I0802 17:49:38.071402       1 server.go:113] FLAG: --domain="cluster.local."
I0802 17:49:38.071406       1 server.go:113] FLAG: --federations=""
I0802 17:49:38.071410       1 server.go:113] FLAG: --healthz-port="8081"
I0802 17:49:38.071413       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I0802 17:49:38.071416       1 server.go:113] FLAG: --kube-master-url=""
I0802 17:49:38.071420       1 server.go:113] FLAG: --kubecfg-file=""
I0802 17:49:38.071422       1 server.go:113] FLAG: --log-backtrace-at=":0"
I0802 17:49:38.071428       1 server.go:113] FLAG: --log-dir=""
I0802 17:49:38.071433       1 server.go:113] FLAG: --log-flush-frequency="5s"
I0802 17:49:38.071440       1 server.go:113] FLAG: --logtostderr="true"
I0802 17:49:38.071445       1 server.go:113] FLAG: --nameservers=""
I0802 17:49:38.071452       1 server.go:113] FLAG: --stderrthreshold="2"
I0802 17:49:38.071457       1 server.go:113] FLAG: --v="2"
I0802 17:49:38.071464       1 server.go:113] FLAG: --version="false"
I0802 17:49:38.071474       1 server.go:113] FLAG: --vmodule=""
I0802 17:49:38.071525       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I0802 17:49:38.071749       1 server.go:198] Skydns metrics enabled (/metrics:10055)
I0802 17:49:38.071757       1 dns.go:147] Starting endpointsController
I0802 17:49:38.071761       1 dns.go:150] Starting serviceController
I0802 17:49:38.071836       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0802 17:49:38.071855       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0802 17:49:38.082493       1 sync_configmap.go:107] ConfigMap kube-system:kube-dns was created
I0802 17:49:38.581981       1 dns.go:171] Initialized services and endpoints from apiserver
I0802 17:49:38.582016       1 server.go:129] Setting up Healthz Handler (/readiness)
I0802 17:49:38.582031       1 server.go:134] Setting up cache handler (/cache)
I0802 17:49:38.582045       1 server.go:120] Status HTTP port 8081

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 6
  • Comments: 18 (5 by maintainers)

Commits related to this issue

Most upvoted comments

It looks like DNS inside busybox does not work properly. At least it works for me with busybox images <= 1.28.4

+216
gogene on Aug 9, 2018

Any solution on this? Also having this problem

+61
qgicup on Aug 7, 2018

In my case it was a missing IP tables rule on a dedicated server. Resolved by executing on the server:

iptables -w -P FORWARD ACCEPT
+21
cgebe on May 27, 2020

@gogene P.S. In 2020.08, the 1.32.0 still has problem in nslookup. (2 years has passed…)

+18
fzyzcjy on Aug 9, 2020

@gogene Ok - version 1.28.4 solves it, works like a charm, thank you, I think we can close this issue.`

$ kubectl -n default exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

Btw. do you know why it can’t resolve svc.cluster.local?

kubectl -n default exec -ti busybox -- nslookup svc.cluster.local
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'svc.cluster.local'
command terminated with exit code 1
+18
cparjaszewski on Aug 9, 2018

Actually the same problem is on newest MacOS, when I run busybox pod and nslookup the domain:

** Can't find kubernetes.default: ***
** Can't find cluster.local: ***
+12
cparjaszewski on Aug 3, 2018

there are two reason caused this issue:

  1. the nslookup program with busybox:latest can not extract the record from dns response udp packet( the correct ip response from dns server)
  2. options ndots:5 in the /etc/resolv.conf caused some other issue when the domain with many dot. after i changed the ndots from 5 to 7, then i can get:
/ # nslookup -debug -timeout=2 mysql-0.mysql.default.svc.cluster.local. 
Server:		10.68.0.2
Address:	10.68.0.2:53

Query #0 completed in 1ms:
Name:	mysql-0.mysql.default.svc.cluster.local
Address: 172.20.185.197

*** Can't find mysql-0.mysql.default.svc.cluster.local.: No answer

/ # cat /etc/resolv.conf 
nameserver 10.68.0.2
search default.svc.cluster.local. svc.cluster.local. cluster.local.
options ndots:7
+3
plantegg on Sep 17, 2020

Try querying one of the the kube-dns pods directly, to see if it’s a network layer issue… e.g.

kubectl -n default exec -ti busybox nslookup kubernetes.default <ip-address-of-pod>

+3
chrisohaver on Aug 3, 2018

On my MacOS I have only one pod with kube-dns:

$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name:

pod/kube-dns-86f4d74b45-b4dd8

On my server (Ubuntu 18.04) I have 3 pods: $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name:

pod/coredns-78fcdf6894-c4sk8
pod/coredns-78fcdf6894-mzv9t
pod/kube-dns-569b8c4c96-bwwvm

$ kubectl -n default exec -ti busybox nslookup kubernetes.default 10.244.0.84:

Server:     10.244.0.84
Address:    10.244.0.84:53

** server can't find kubernetes.default: NXDOMAIN

*** Can't find kubernetes.default: No answer

$ kubectl -n default exec -ti busybox nslookup cluster.local 10.244.0.84:

Server:     10.244.0.84
Address:    10.244.0.84:53


*** Can't find cluster.local: No answer

$ kubectl -n default exec -ti busybox nslookup kubernetes.default 10.244.0.82:

Server:     10.244.0.82
Address:    10.244.0.82:53

** server can't find kubernetes.default: NXDOMAIN

*** Can't find kubernetes.default: No answer

$ kubectl -n default exec -ti busybox nslookup cluster.local 10.244.0.82:

Server:     10.244.0.82
Address:    10.244.0.82:53


*** Can't find cluster.local: No answer

More details on these 2 coredns pods on Ubuntu:

$ kubectl -n kube-system describe pod coredns-78fcdf6894-c4sk8:

Name:               coredns-78fcdf6894-c4sk8
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               kubernetes-slave/37.59.16.40
Start Time:         Mon, 23 Jul 2018 13:56:35 +0000
Labels:             k8s-app=kube-dns
                    pod-template-hash=3497892450
Annotations:        <none>
Status:             Running
IP:                 10.244.0.84
Controlled By:      ReplicaSet/coredns-78fcdf6894
Containers:
  coredns:
    Container ID:  docker://e76a47934a878a44158d8fa90bc3c0077fa3f11f8c82eb3c62f9615f24e76337
    Image:         k8s.gcr.io/coredns:1.1.3
    Image ID:      docker-pullable://k8s.gcr.io/coredns@sha256:db2bf53126ed1c761d5a41f24a1b82a461c85f736ff6e90542e9522be4757848
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Wed, 01 Aug 2018 14:25:51 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 01 Aug 2018 14:24:59 +0000
      Finished:     Wed, 01 Aug 2018 14:25:50 +0000
    Ready:          True
    Restart Count:  2
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-ch7j7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-ch7j7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-ch7j7
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From                       Message
  ----     ------            ----                 ----                       -------
  Warning  DNSConfigForming  59s (x9409 over 8d)  kubelet, kubernetes-slave  Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.125.211.1 213.186.33.99 127.0.0.1

$ kubectl -n kube-system describe pod coredns-78fcdf6894-mzv9t:

Name:               coredns-78fcdf6894-mzv9t
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               kubernetes-slave/37.59.16.40
Start Time:         Mon, 23 Jul 2018 13:56:35 +0000
Labels:             k8s-app=kube-dns
                    pod-template-hash=3497892450
Annotations:        <none>
Status:             Running
IP:                 10.244.0.82
Controlled By:      ReplicaSet/coredns-78fcdf6894
Containers:
  coredns:
    Container ID:  docker://03aee6a3ae008ddbe14ec6ad14190ab9bc6e6e20ff492d4d3cd903cd55d89bf1
    Image:         k8s.gcr.io/coredns:1.1.3
    Image ID:      docker-pullable://k8s.gcr.io/coredns@sha256:db2bf53126ed1c761d5a41f24a1b82a461c85f736ff6e90542e9522be4757848
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Wed, 01 Aug 2018 14:25:23 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 23 Jul 2018 13:56:38 +0000
      Finished:     Wed, 01 Aug 2018 14:24:56 +0000
    Ready:          True
    Restart Count:  1
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-ch7j7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-ch7j7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-ch7j7
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From                       Message
  ----     ------            ----                ----                       -------
  Warning  DNSConfigForming  2m (x9382 over 8d)  kubelet, kubernetes-slave  Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.125.211.1 213.186.33.99 127.0.0.1
+2
cparjaszewski on Aug 3, 2018

Just for the records, I opened a new issue at the bugtracker of busybox: https://bugs.busybox.net/show_bug.cgi?id=14671

+1
guettli on Mar 16, 2022
Read more comments on GitHub
← kubernetes: dnf/yum/rpm installation of Kubernetes packages from official repos fails on RHEL 8 systems with FIPS enabled
kubernetes: DNS intermittent delays of 5s →