kubernetes: Networking: Pods are unable to communicate with: other pods, the API Server and the outside world

What happened:

On a freshly bootstrapped cluster the Pods cannot reach neither other pods, nor the API Server, nor the outside world

What you expected to happen:

Be able to reach the API Server as well as the outside world

How to reproduce it (as minimally and precisely as possible):

I tried with the following two variants. Result is the same.

1: NanoPi m4 as Master. Wife Connected to my Cable Modem. eth0 connected to separate switch for the node network. Nodes are Raspberry PI3 with Debian Buster and one amd64 based node. Physical Network is as follows: Internet <–> Cable Modem <–> 192.168.178.0/24 <–> (WiFi) NanoPi m4 (LAN )<–> 172.16.0.0/24 Switch <–> Nodes NAT Routing works from the Nodes to the Internet. Docker CA Install according to docker website Kubernetes Install according to Kubernetes Website kubeadm init --apiserver-advertise-address=172.16.0.1 --pod-network-cidr=10.244.0.0/16 sysctl net.bridge.bridge-nf-call-iptables=1 kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml kubeadm join 172.16.0.1:6443 --token kkc1c7.hm4r65gbyf4g059a --discovery-token-ca-cert-hash sha256:dc04b50bd56251c2c2f2b315ad6dcca9ddf0ffda964e3f3ef63fd6a7d7800a75

2: Virtual Box VMs Network is similar to above: Internet <–> Cable Modem <–> 192.168.178.0/24 <–> (WiFi) Laptop <–> (Bridged Network) VBox Master VM <–> 172.16.0.0/24 Virtual Network “k8s” <–> Node VM Networking on the VMs works as expected. Same steps as above to set up the cluster.

kubectl run alpine --image=alpine -it --restart=Never -- sh
/ # ping google.de
ping: bad address 'google.de'

Ping Test fails for various Cluster IPs shown in `env` as well. Only the gateway IP is pingable
/ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether 0a:58:0a:f4:01:08 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.8/24 scope global eth0
       valid_lft forever preferred_lft forever
/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.244.1.1      0.0.0.0         UG    0      0        0 eth0
10.244.0.0      10.244.1.1      255.255.0.0     UG    0      0        0 eth0
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
/ # ping 10.244.1.1
PING 10.244.1.1 (10.244.1.1): 56 data bytes
64 bytes from 10.244.1.1: seq=0 ttl=64 time=0.193 ms
64 bytes from 10.244.1.1: seq=1 ttl=64 time=0.173 ms
^C
--- 10.244.1.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.173/0.183/0.193 ms

Anything else we need to know?:

At first I ran into the issue trying to use helm, where helm ls resulted in a timeout. Hence I tried to run the above mentioned alpine pod in the default namespace, as well as in kube-system. Below is some testing where alpine runs in kube-system

kubectl run alpine --image=alpine -it -n kube-system --restart=Never -- sh
If you don't see a command prompt, try pressing enter.

/ # nslookup google.de
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'google.de': Try again
/ # cat /etc/resolv.conf 
nameserver 10.96.0.10
search kube-system.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/ # env
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT=443
KUBE_DNS_SERVICE_PORT_DNS_TCP=53
HOSTNAME=alpine
TILLER_DEPLOY_SERVICE_HOST=10.103.53.185
SHLVL=1
HOME=/root
KUBE_DNS_SERVICE_HOST=10.96.0.10
TILLER_DEPLOY_SERVICE_PORT=44134
TILLER_DEPLOY_PORT=tcp://10.103.53.185:44134
TILLER_DEPLOY_PORT_44134_TCP_ADDR=10.103.53.185
KUBE_DNS_PORT=udp://10.96.0.10:53
TILLER_DEPLOY_PORT_44134_TCP_PORT=44134
KUBE_DNS_SERVICE_PORT=53
TILLER_DEPLOY_PORT_44134_TCP_PROTO=tcp
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
TILLER_DEPLOY_SERVICE_PORT_TILLER=44134
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBE_DNS_PORT_53_TCP_ADDR=10.96.0.10
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBE_DNS_PORT_53_UDP_ADDR=10.96.0.10
TILLER_DEPLOY_PORT_44134_TCP=tcp://10.103.53.185:44134
KUBE_DNS_PORT_53_TCP_PORT=53
KUBE_DNS_PORT_53_TCP_PROTO=tcp
KUBE_DNS_PORT_53_UDP_PORT=53
KUBE_DNS_SERVICE_PORT_DNS=53
KUBE_DNS_PORT_53_UDP_PROTO=udp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_HOST=10.96.0.1
PWD=/
KUBE_DNS_PORT_53_TCP=tcp://10.96.0.10:53
KUBE_DNS_PORT_53_UDP=udp://10.96.0.10:53
/ # ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10): 56 data bytes
^C
--- 10.96.0.10 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/ # ping 10.96.0.1
PING 10.96.0.1 (10.96.0.1): 56 data bytes
^C
--- 10.96.0.1 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
/ # ping 10.103.53.185
PING 10.103.53.185 (10.103.53.185): 56 data bytes
^C
--- 10.103.53.185 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss

Bringing up a 2nd instance of alpine and trying to ping it:

/ # ping 10.244.1.10
PING 10.244.1.10 (10.244.1.10): 56 data bytes
^C
--- 10.244.1.10 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.1”, GitCommit:“eec55b9ba98609a46fee712359c7b5b365bdd920”, GitTreeState:“clean”, BuildDate:“2018-12-13T10:39:04Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/arm64”} Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.1”, GitCommit:“eec55b9ba98609a46fee712359c7b5b365bdd920”, GitTreeState:“clean”, BuildDate:“2018-12-13T10:31:33Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/arm64”}

• OS (e.g. from /etc/os-release): NanoPi m4 is Armbian stretch. All others, both physical and VMs are Debian Buster

I hope the information provided is sufficient. If not, please let me know and I will provide additional information. You can also reach me on slack @Vonor to do some live testing.

/kind bug