kubernetes: kubeadm 1.4: service IPs not reachable, kube-dns not resolving names

Hello @kubernetes/sig-cluster-lifecycle!

Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“4”, GitVersion:“v1.4.0”, GitCommit:“a16c0a7f71a6f93c7e0f222d961f4675cd97a46b”, GitTreeState:“clean”, BuildDate:“2016-09-26T18:16:57Z”, GoVersion:“go1.6.3”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“4”, GitVersion:“v1.4.0”, GitCommit:“a16c0a7f71a6f93c7e0f222d961f4675cd97a46b”, GitTreeState:“clean”, BuildDate:“2016-09-26T18:10:32Z”, GoVersion:“go1.6.3”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

• Cloud provider or hardware configuration: Non-cloud VMs in Hyper-V environment, 4Gb RAM, 2 CPUs each.
• OS (e.g. from /etc/os-release):

CentOS Linux release 7.2.1511 (Core)

• Kernel (e.g. uname -a): Linux kube-master 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• See Also This issue inspired our research. 33671#issuecomment-250477571

What happened: We used kubeadm and the procedure in Installing Kubernetes on Linux with kubeadm and for the most part the installation went well. We did not enable the master to also be a minion node.

The first time the services were started, the kube-dns containers took at least 10 minutes to start up. But everything looked ok.

Following along the debugging services document:

[root@kube-node2 ~]# docker ps
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS               NAMES
10944fbe061d        busybox                                                      "sleep 3600"             2 minutes ago       Up 2 minutes                           k8s_busybox.b07d13cc_busybox_default_7e48bc99-8673-11e6-980f-00155d69014b_fef430abf6

First I get info about the hostname and DNS services

[root@kube-master ~]# kc get svc
NAME         CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
hostnames    100.68.31.201    <none>        80/TCP      27m

[root@kube-master ~]# kc get svc --namespace=kube-system
NAME                   CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kube-dns               100.64.0.10    <none>        53/UDP,53/TCP   6h

What about the DNS service?

[root@kube-master ~]# kd  svc --namespace=kube-system
Name:                   kube-dns
Namespace:              kube-system
Labels:                 component=kube-dns ...
Selector:               name=kube-dns
Type:                   ClusterIP
IP:                     100.64.0.10
Port:                   dns     53/UDP
Endpoints:              10.32.0.4:53
Port:                   dns-tcp 53/TCP
Endpoints:              10.32.0.4:53

I enter the busybox container

[root@kube-node2 ~]# docker exec -it 109 sh

And try to lookup the hostnames name using the default DNS server, the ClusterIP of the DNS

/ # nslookup hostnames
Server:    100.64.0.10
Address 1: 100.64.0.10
nslookup: can't resolve 'hostnames'

This command takes 30 seconds to fail. Smells like a timeout.

Now I use the DNS node ClusterIP as the DNS server.

/ # nslookup hostnames 10.32.0.4
Server:    10.32.0.4
Address 1: 10.32.0.4 kube-dns-2247936740-asua6

Name:      hostnames
Address 1: 100.68.31.201 hostnames.default.svc.cluster.local

And we get 100.68.31.201, the ClusterIP of the hostnames service.

What you expected to happen: I expect the nslookup hostnames command to return something like:

/ # nslookup hostnames
Server:    ?.?.?.?
Address 1: ?.?.?.? kube-dns-2247936740-asua6

Name:      hostnames
Address 1: 100.68.31.201 hostnames.default.svc.cluster.local

Service access

We are able to access the service from the nodes:

[root@kube-node1 ~]# curl -s 100.68.31.201
hostnames-3799501552-2zsfe

[root@kube-node2 ~]#  curl -s 100.68.31.201
hostnames-3799501552-2zsfe

But not from the busybox container:

[root@kube-node2 ~]# docker exec -it 695f74c480ae sh
/ # wget 100.66.122.211:80
Connecting to 100.66.122.211:80 (100.66.122.211:80)
[hang]

Anything else do we need to know:

Per the referenced issue, here is an empty KUBE-SERVICES chain

[root@kube-node2 ~]# iptables -L KUBE-SERVICES
Chain KUBE-SERVICES (1 references)
target     prot opt source               destination

I think the commenter is having the same issue we are.

Here is the iptables dump from node2, edited to remove everything other than kube-system/kube-dns:dns and default/hostnames.

[root@kube-node2 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Thu Sep 29 17:41:25 2016
*mangle
:PREROUTING ACCEPT [494013:166892600]
:INPUT ACCEPT [207403:117295288]
:FORWARD ACCEPT [9184:5767521]
:OUTPUT ACCEPT [114336:43871628]
:POSTROUTING ACCEPT [123463:49637597]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Thu Sep 29 17:41:25 2016
# Generated by iptables-save v1.4.21 on Thu Sep 29 17:41:25 2016
*nat
:PREROUTING ACCEPT [6:966]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-2HD6WT2OGWLMDWEX - [0:0]
:KUBE-SEP-726LBT3CKELEA4BM - [0:0]
:KUBE-SEP-ES652KEXJGQ2TLCD - [0:0]
:KUBE-SEP-U5YZACCZAGQCTFI3 - [0:0]
:KUBE-SEP-ZT5TVM6PMFDFQAMO - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NWV5X2332I4OT4T3 - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -j WEAVE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-2HD6WT2OGWLMDWEX -s 10.40.0.5/32 -m comment --comment "default/hostnames:" -j KUBE-MARK-MASQ
-A KUBE-SEP-2HD6WT2OGWLMDWEX -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.40.0.5:9376
-A KUBE-SEP-726LBT3CKELEA4BM -s 10.46.0.7/32 -m comment --comment "default/hostnames:" -j KUBE-MARK-MASQ
-A KUBE-SEP-726LBT3CKELEA4BM -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.46.0.7:9376
-A KUBE-SEP-ES652KEXJGQ2TLCD -s 10.32.0.4/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-ES652KEXJGQ2TLCD -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.32.0.4:53
-A KUBE-SEP-U5YZACCZAGQCTFI3 -s 10.40.0.6/32 -m comment --comment "default/hostnames:" -j KUBE-MARK-MASQ
-A KUBE-SEP-U5YZACCZAGQCTFI3 -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.40.0.6:9376
-A KUBE-SEP-ZT5TVM6PMFDFQAMO -s 10.32.0.4/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZT5TVM6PMFDFQAMO -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.32.0.4:53
-A KUBE-SERVICES -d 100.68.31.201/32 -p tcp -m comment --comment "default/hostnames: cluster IP" -m tcp --dport 80 -j KUBE-SVC-NWV5X2332I4OT4T3
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 100.64.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-ZT5TVM6PMFDFQAMO
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-2HD6WT2OGWLMDWEX
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-U5YZACCZAGQCTFI3
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -j KUBE-SEP-726LBT3CKELEA4BM
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-ES652KEXJGQ2TLCD
-A WEAVE -s 10.32.0.0/12 -d 224.0.0.0/4 -j RETURN
-A WEAVE ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j MASQUERADE
-A WEAVE -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j MASQUERADE
COMMIT
# Completed on Thu Sep 29 17:41:25 2016
# Generated by iptables-save v1.4.21 on Thu Sep 29 17:41:25 2016
*filter
:INPUT ACCEPT [9:2378]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [7:3454]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -d 172.17.0.1/32 -i docker0 -p tcp -m tcp --dport 6783 -j DROP
-A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6783 -j DROP
-A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6784 -j DROP
-A INPUT -i docker0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i docker0 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -i docker0 -o weave -j DROP
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -o weave -j WEAVE-NPC
-A FORWARD -o weave -j LOG --log-prefix "WEAVE-NPC:"
-A FORWARD -o weave -j DROP
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]<@p dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
COMMIT
# Completed on Thu Sep 29 17:41:25 2016