kubernetes: Coredns service do not work,but endpoint is ok,the other SVCs are normal only except dns

here is the status right now:

root:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

root:~# kubectl get svc -n kube-system | grep dns
kube-dns               ClusterIP   10.96.0.10       <none>         53/UDP,53/TCP   68d

root:~# kubectl get pod -n kube-system -o wide | grep dns 
coredns-657fc9d5b8-85mfb                1/1       Running     0          28m       172.26.2.164   yksp009029

root:~# kubectl get ep kube-dns -n kube-system
NAME       ENDPOINTS                         AGE
kube-dns   172.26.2.164:53,172.26.2.164:53   68d


#here are svcs info
root:~# kubectl get svc -o wide -n kube-system
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP    PORT(S)         AGE       SELECTOR
heapster               ClusterIP   10.111.110.137   <none>         80/TCP          67d       k8s-app=heapster
kube-dns               ClusterIP   10.96.0.10       <none>         53/UDP,53/TCP   68d       k8s-app=kube-dns
kubernetes-dashboard   NodePort    10.104.130.140   <none>         443:30843/TCP   67d       k8s-app=kubernetes-dashboard
monitoring-grafana     ClusterIP   10.109.146.115   192.168.9.60   80/TCP          67d       k8s-app=grafana
monitoring-influxdb    ClusterIP   10.101.110.69    <none>         8086/TCP        67d       k8s-app=influxdb
traefik-web-ui         ClusterIP   10.98.101.187    <none>         80/TCP          67d       k8s-app=traefik-ingress-lb


#all SVCs work normally except kube-dns
root:~# telnet 10.111.110.137 80
Trying 10.111.110.137...
Connected to 10.111.110.137.
Escape character is '^]'.
^C
Connection closed by foreign host.

#The port of the kube-dns svc is in a non-listening state,but in fact, the pod on end-point is listening
root:~# telnet 10.96.0.10 53
Trying 10.96.0.10...
telnet: Unable to connect to remote host: No route to host

#ping is ok
root:~# ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10) 56(84) bytes of data.
64 bytes from 10.96.0.10: icmp_seq=1 ttl=64 time=0.055 ms
64 bytes from 10.96.0.10: icmp_seq=2 ttl=64 time=0.038 ms

#the kube-dns endpoint pod is in listening state
root:~# telnet 172.26.2.164 53
Trying 172.26.2.164...
Connected to 172.26.2.164.
Escape character is '^]'.
Connection closed by foreign host.

It just look like every thing is fine.Whether it is viewing the svc/pod/endpoint status, logs,iptables,network,everything is normal. But only the one svc–kube-dns don’t work,it causes to be unable to parse the domain name in pod,but ping IP is ok. So worry about this for a long time,Somebody can help me?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 23 (9 by maintainers)

Most upvoted comments

Same issue, I also had to redeploy the coredns deployment:

$ kubectl get deploy -n kube-system
NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
coredns                2         2         2            2           29h
$ wget https://raw.githubusercontent.com/zlabjp/kubernetes-scripts/master/force-update-deployment
$ chmod +x force-update-deployment
$ force-update-deployment coredns -n kube-system

@chrisohaver After I redeployed CoreDNS twice, the problem was solved, which was very strange, but so far the problem has not recurred.I will close this issue,thank you so much!