kubernetes: Pods cant access ServiceIPs/DNS in a fresh setup with kubeadm on ubuntu 16.04

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): Yes, I have looked at and followed https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ and https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#does-the-service-work-by-ip to no avail

What keywords did you search in Kubernetes issues before filing this one?

kubernetes dns failure, kubernetes pods cant acces service IP

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: Bare Metal via OVH
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a): Linux k8s-06.elenty.com 4.9.0-040900-generic #201612111631 SMP Sun Dec 11 21:33:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Kubeadm
• Others: Flannel

What happened:

Kube DNS is running

[root@k8s-06 ~]$ kubectl get po --all-namespaces | grep dns
kube-system   kube-dns-3913472980-170fr                   3/3       Running   12         47m

Kube DNS has a service IP

[root@k8s-06 ~]$ kubectl get svc kube-dns -n kube-system
NAME       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   10.96.0.10   <none>        53/UDP,53/TCP   46m

Kube DNS works from the node itself

[root@k8s-06 ~]$ dig google.com @10.96.0.10 +short
216.58.217.174

Kube DNS works inside a normal ubuntu docker container

[root@k8s-06 ~]$ docker run -ti ubuntu:latest
root@601fa1fe4dfe:/#  apt update && apt install dnsutils
root@601fa1fe4dfe:/# dig google.com @10.96.0.10 +short
216.58.217.174

Kube DNS fails inside a pod

[root@k8s-06 ~]$ kubectl describe po kube-dns-3913472980-170fr -n kube-system | grep IP
IP:		10.244.0.11
[root@k8s-06 ~]$ cat ubuntu.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  namespace: default
spec:
  containers:
  - image: ubuntu
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always
[root@k8s-06 ~]$ kubectl create -f ubuntu.yaml 
[root@k8s-06 ~]$ kubectl exec -ti ubuntu bash
root@ubuntu:/# apt update && apt install -y dnsutils
0% [Connecting to archive.ubuntu.com] [Connecting to security.ubuntu.com]^C
<hangs>
root@ubuntu:/# cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local ovh.net
options ndots:5
root@ubuntu:/# echo "nameserver 8.8.8.8" > /etc/resolv.conf
root@ubuntu:/# apt update && apt install dnsutils
<works> … <snip>
root@ubuntu:/# dig google.com @10.244.0.11 +short
216.58.217.174
root@ubuntu:/# dig google.com @10.96.0.10 +short 
<hangs>

Iptables output:

[root@k8s-06 ~]$ iptables-save | grep dns
-A KUBE-SEP-A7YQ4MY4TZII3JTK -s 10.244.0.11/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-A7YQ4MY4TZII3JTK -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.11:53
-A KUBE-SEP-VR6VIIG2A6524KLY -s 10.244.0.11/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-VR6VIIG2A6524KLY -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.11:53
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-A7YQ4MY4TZII3JTK
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-VR6VIIG2A6524KLY

What you expected to happen: This command (ubuntu is a pod) should not hang:

<hangs>

How to reproduce it (as minimally and precisely as possible): We are setting up kubenertes via kubeadm inside some ansible scripts and can reproduce the issue everytime. We setup a cluster basically with these steps:

1. install docker from apt and kubernetes from the ppa http://apt.kubernetes.io/
2. Run kube-adm specifiying the token, apiserver-advertise-address and pod-network-cidr on the command line
3. Setup flannel from the flannel github page
4. Run the above commands, note the inability to access the service IP

Anything else we need to know: I’m on slack as @zachg in the kubernetes group, feel free to ping me there for realtime back and fourth if that’s better than here on github.