kubernetes: kubedns nslookup: can't resolve 'kubernetes.default'

I had launched kubedns , but it can’t resolve domain name. Maybe there are some miss in my environment, can you help me fetch out!

my launched kubedns

[root@wlan-cloudserver31 dns]# kubectl get  services --all-namespaces -o wide
NAMESPACE     NAME                  LABELS                                                                           SELECTOR             IP(S)            PORT(S)
default       kubernetes            component=apiserver,provider=kubernetes                                          <none>               10.254.0.1       443/TCP
kube-system   heapster              kubernetes.io/cluster-service=true,kubernetes.io/name=Heapster                   k8s-app=heapster     10.254.189.105   8082/TCP
kube-system   kube-dns              k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS   k8s-app=kube-dns     10.254.237.18    53/UDP
                                                                                                                                                           53/TCP
kube-system   monitoring-grafana    kubernetes.io/cluster-service=true,kubernetes.io/name=monitoring-grafana         name=influxGrafana   10.254.225.111   3000/TCP
kube-system   monitoring-influxdb   kubernetes.io/cluster-service=true,kubernetes.io/name=monitoring-influxdb        name=influxGrafana   10.254.10.95     8083/TCP
                                                                                                                                                           8086/TCP
[root@wlan-cloudserver31 dns]# kubectl get  pods --all-namespaces -o wide
NAMESPACE     NAME                     READY     STATUS    RESTARTS   AGE       NODE
default       busybox                  1/1       Running   2          19h       172.27.8.211
default       redis-master-rze58       1/1       Running   2          2d        172.27.8.214
kube-system   heapster-zcd76           1/1       Running   0          15h       172.27.8.214
kube-system   influxdb-grafana-h9rnx   2/2       Running   0          1d        172.27.8.214
kube-system   kube-dns-v9-oaep5        4/4       Running   2          19h       172.27.8.214

[root@wlan-cloudserver31 dns]# kubectl get nodes
NAME           LABELS                                STATUS
172.27.8.211   kubernetes.io/hostname=172.27.8.211   Ready
172.27.8.212   kubernetes.io/hostname=172.27.8.212   NotReady
172.27.8.214   kubernetes.io/hostname=172.27.8.214   Ready

can’t resolve kubernetes.default

[root@wlan-cloudserver31 dns]# kubectl exec busybox -- nslookup kubernetes.default
Server:    10.72.66.36
Address 1: 10.72.66.36 ns2.h3c.com

nslookup: can't resolve 'kubernetes.default'
error: Error executing remote command: Error executing command in container: Error executing in Docker Container: 1
[root@wlan-cloudserver31 dns]#

there are some logs for `skydns` container

In the last 19 hours, container have to restart 2 times. the logs seems a little less!

[root@wlan-cloudserver34 wlanuser]# docker ps -a
CONTAINER ID        IMAGE                                            COMMAND                CREATED             STATUS                   PORTS               NAMES
a98f9192ae61        gcr.io/google_containers/skydns:2015-03-11-001   "/skydns -machines=h   2 hours ago         Up 2 hours                                   k8s_skydns.380e6cd5_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_41f5ecf6          
31aa3417cc21        gcr.io/google_containers/skydns:2015-03-11-001   "/skydns -machines=h   8 hours ago         Exited (2) 2 hours ago                       k8s_skydns.380e6cd5_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_81b01cbf          
e5e0455cf99c        kubernetes/heapster:canary                       "/heapster --vmodule   15 hours ago        Up 15 hours                                  k8s_heapster.3a2d03b7_heapster-zcd76_kube-system_3a99af72-82ea-11e5-a8b4-fa163e77e286_612b5665           
7d7cc702f4e1        gcr.io/google_containers/pause:0.8.0             "/pause"               15 hours ago        Up 15 hours                                  k8s_POD.e4cc795_heapster-zcd76_kube-system_3a99af72-82ea-11e5-a8b4-fa163e77e286_96e08804                 
6432d0de5ecb        gcr.io/google_containers/exechealthz:1.0         "/exechealthz '-cmd=   19 hours ago        Up 19 hours                                  k8s_healthz.fa8774e2_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_5bcaec1e         
77ef14d19636        gcr.io/google_containers/skydns:2015-03-11-001   "/skydns -machines=h   19 hours ago        Exited (2) 8 hours ago                       k8s_skydns.380e6cd5_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_df1721d9          
14d5f5001861        gcr.io/google_containers/kube2sky:1.11           "/kube2sky -domain=c   19 hours ago        Up 19 hours                                  k8s_kube2sky.14894b68_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_77e973ac        
8ecd390d8810        gcr.io/google_containers/etcd:2.0.9              "/usr/local/bin/etcd   19 hours ago        Up 19 hours                                  k8s_etcd.a0988a50_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_cdbe887c            
a1f61baa1b0c        gcr.io/google_containers/pause:0.8.0             "/pause"               19 hours ago        Up 19 hours                                  k8s_POD.2688308a_kube-dns-v9-oaep5_kube-system_06e1de3d-82c2-11e5-a8b4-fa163e77e286_d177494a             
c029e1a8f304        kubernetes/heapster_grafana:v2.1.0               "/bin/sh -c /run.sh"   23 hours ago        Up 23 hours                                  k8s_grafana.c8bbb6fb_influxdb-grafana-h9rnx_kube-system_7801146f-81d8-11e5-a8b4-fa163e77e286_9256222e    
29f245a4de02        kubernetes/heapster_influxdb:v0.5                "influxd --config /e   23 hours ago        Up 23 hours                                  k8s_influxdb.5fc13a9e_influxdb-grafana-h9rnx_kube-system_7801146f-81d8-11e5-a8b4-fa163e77e286_b0db77ce   
f78cbec5800e        redis                                            "/entrypoint.sh redi   23 hours ago        Up 23 hours                                  k8s_master.1681ebfb_redis-master-rze58_default_92a91592-8138-11e5-a8b4-fa163e77e286_42c8ee9c             
91f64ba7ce1b        gcr.io/google_containers/pause:0.8.0             "/pause"               23 hours ago        Up 23 hours                                  k8s_POD.ac4f2d56_influxdb-grafana-h9rnx_kube-system_7801146f-81d8-11e5-a8b4-fa163e77e286_996e425d        
c9928e690995        gcr.io/google_containers/pause:0.8.0             "/pause"               23 hours ago        Up 23 hours                                  k8s_POD.49eee8c2_redis-master-rze58_default_92a91592-8138-11e5-a8b4-fa163e77e286_10493560                
[root@wlan-cloudserver34 wlanuser]# docker logs 31aa3417cc21
2015/11/04 18:02:35 skydns: falling back to default configuration, could not read from etcd: 100: Key not found (/skydns/config) [567]
2015/11/04 18:02:35 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
2015/11/04 18:02:35 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]
[root@wlan-cloudserver34 wlanuser]# docker logs 77ef14d19636
2015/11/04 07:02:10 skydns: falling back to default configuration, could not read from etcd: 100: Key not found (/skydns) [1]
2015/11/04 07:02:10 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
2015/11/04 07:02:10 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]
[root@wlan-cloudserver34 wlanuser]# docker logs a98f9192ae61
2015/11/05 00:02:46 skydns: falling back to default configuration, could not read from etcd: 100: Key not found (/skydns/config) [867]
2015/11/05 00:02:46 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
2015/11/05 00:02:46 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]
2015/11/05 02:11:00 skydns: failure to forward request "read udp 10.72.66.37:53: i/o timeout"
2015/11/05 02:11:04 skydns: failure to forward request "read udp 10.72.66.36:53: i/o timeout"
[root@wlan-cloudserver34 wlanuser]#

for the last skydns , it hint failure to forward request ..., how does it happen? my config is 0.0.0.0:53.

skydns-rc.yaml

[root@wlan-cloudserver31 dns]# more  skydns-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v9
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    version: v9
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v9
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v9
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: gcr.io/google_containers/etcd:2.0.9
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        command:
        - /usr/local/bin/etcd
       # - --privileged=true
       # - -data-dir
       # - /home/data/etcd
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - mountPath: /home/data/etcd
          name: etcd-storage
      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.11
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/kube2sky"
        - -domain=cluster.local
       # - -etcd-server=http://127.0.0.1:4001
        - -kube_master_url=http://172.27.8.210:8080
      - name: skydns
        image: gcr.io/google_containers/skydns:2015-03-11-001
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://127.0.0.1:2379
        - -addr=0.0.0.0:53
       # - -ns-rotate=false
        - -domain=cluster.local
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 1
          timeoutSeconds: 5
      - name: healthz
        image: gcr.io/google_containers/exechealthz:1.0
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 47 (22 by maintainers)

Most upvoted comments

If skydns is using 127.0.1.1, that is because your node is using 127.0.1.1. In such cases, you need to start kubelet with the resolv-conf flag that points to a custom resolv.conf file, and specify the nodeip as the nameserver in that custom resolv.conf file. That is the kubernetes-supported approach to fixing these kind of issues where node has dnsmasq process running.

ArtfulCoder on Apr 9, 2016