kubernetes: Kube-DNS Name does not resolve.

Can we try the following

1. Test if service selector are working

kubectl get endpoints kube-dns

if this fails, i would look if service controller/controller manager are running correctly on the cluster. You should find (usually 2) endpoints, do they match your DNS pod ips?

1. does the pod actually work?

kubectl exec --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns -- nslookup google.com 127.0.0.1

kubectl exec --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns -- nslookup kubernetes 127.0.0.1

Success on the first call means that dns can connect to your upstream dns server. success on the second means DNS itself is working and it is a network path to DNS server. if 1st failed check network path to upstream, if second failed then you should get health prop failures (do you get them?)

1. Test if your DNS pod is reachable from nodes (1 node + the node that has curl-775f9567b5-ktcf5 pod)

nslookup google.com 10.96.0.10

if not successful then your service (kube-proxy working? not hung?) or pod network routing is not working as expected. To test pod network try deploying nginx and curl it from node and/or a pod (using IP).

@cjbottaro & @parvez84, may I ask did y’all resolve this?

@boiside how u resolve it with coredns?

Has anyone any further troubleshooting steps for this? My DNS pod is healthy and other pods reach it fine, I can resolve external names but nothing under svc.cluster.local. The 2nd command mentioned above fails for me (the 1st succeeds):

... nslookup kubernetes 127.0.0.1
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost

nslookup: can't resolve 'kubernetes': Name does not resolve
command terminated with exit code 1

This only happens intermittently, typically after restarting minikube. Restarting the kube-dns pod doesn’t help. Is there a document anywhere describing by what mechanism Kubernetes services are supposed to end up in the kube-dns tables? I am not even sure what logs to check or what pods to restart…

EDIT: running on the kube-dns pod itself, I find entries for pods but not for services:

/ # nslookup kube-dns-86f4d74b45-l92dn 127.0.0.1
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost

Name:      kube-dns-86f4d74b45-l92dn
Address 1: 172.17.0.17 kube-dns-86f4d74b45-l92dn
/ # nslookup kube-dns.kube-system 127.0.0.1
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost

nslookup: can't resolve 'kube-dns.kube-system': Name does not resolve
/ # nslookup www.google.com 127.0.0.1
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost

Name:      www.google.com
Address 1: 216.58.201.196 ber01s09-in-f196.1e100.net
Address 2: 2a00:1450:4006:808::2004 mrs08s06-in-x04.1e100.net

kubectl get endpoints kube-dns works normally, and the kube-dns log has those entries:

I1023 15:36:24.819513       1 dns.go:146] Starting endpointsController
I1023 15:36:24.819521       1 dns.go:149] Starting serviceController
I1023 15:36:24.825288       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I1023 15:36:24.825300       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I1023 15:36:25.320925       1 dns.go:170] Initialized services and endpoints from apiserver

… I am out of ideas.

Update on our story. It seems we were partially overload Kube-DNS plus there was a bug in the GKE provided DNS server. So updating our Kube-DNS configmap alleviated the issue, then the GKE team supplied a new DNS configuration for their managed services which likely would resolve the issue for us without making the kube-dns changes.

However, I realize this is the Kubernetes repo and anyone who is not on GKE may still benefit from the final, settled kube-dns config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {
      "internal": [ "169.254.169.254" ]
    }
  upstreamNameservers: |
    [ "8.8.8.8", "8.8.4.4" ]

What does this config do? As I understand it after reading the docs and hearing from GKE support, this has only internal names resolve to the internal DNS server 169.254.169.254 (this IP may be different for you if you are not on GKE); everything else resolves to the upstream name servers which are in this case Google’s public name servers. What does this accomplish? It allows kube-dns to quickly determine if the lookup request is internal, and if it is not then send the request upstream. Our company’s code base running in k8s makes a great many DNS queries on a regular basis so this change did help us a lot before GKE team even applied their internal DNS fix.

I am also getting Name or service not known logs inside my application pods when the pods are trying to connect to backends like databases(RDS,Elastic Cache Redis,Ec2 Mongo), I increased the number of pods of kube-dns thinking may be because of load. but no luck. Still seeing those errors in huge number. Created a cluster using kops running with 1.8.8 version

I believe this is due to ndots:5 - try removing that

I got the same issue. I exec into the pod and then remove ndots: 5 or modify to ndots:1 (of file /etc/resolv.conf) => its working, but i don’t know why ? pls someone tell me how to solve this problem (without exec into pod) ?

Solved… my pods were pointing at the DNS server running on the Minikube VM, not at kube-dns, so this is a minikube problem on restart.

As a side note in case that helps someone, changing the “-v=2” arguments in the kube-dns deployment spec is all it takes to see kube-dns actually creating its records as it detects services and pods.

I believe this is due to ndots:5 - try removing that