kubernetes: Kubernetes, can't reach other node services

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened:

I’m playing with Kubernetes/kubeadm inside 3 VirtualBox VMs with CentOS 7, 1 master and 2 minions. Unfortunately installation manuals say something like every service will be accessible from every node, every pod will see all other pods, but I don’t see this happening. I can access the service only from that node where the specific pod runs.

What you expected to happen:

Every service is accessible from every node, every pod can see all other pods.

How to reproduce it (as minimally and precisely as possible):

Use kubeadm to setup 1 master and 2 minions, all with CentOS 7, like here is written: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/, Flannel for networking, I just replaced demo app with simple Nginx pod. Expose the service with default type or NodePort and try to reach it from other VMs and from Host.

Environment:

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**:
• OS: CentOS Linux 7 (Core)
• Kernel: 3.10.0-693.2.2.el7.x86_64
• Install tools: kubeadm
• VirtualBox: 5.1.26 r117224

Anything else we need to know?:

Every VM has 2 adapters: NAT and Host-only. Second one has IPs 10.0.13.101-254.

• Master-1: 10.0.13.104
• Minion-1: 10.0.13.105
• Minion-2: 10.0.13.106

Get all pods from master:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY     STATUS    RESTARTS   AGE
default       busybox                            1/1       Running   12         1d
default       nginx-demo-2867147694-f6f9m        1/1       Running   4          1d
default       nginx-demo2-2631277934-hd7jz       1/1       Running   2          1d
default       nginx-demo3-2046388352-6zj6t       1/1       Running   2          1d
default       nginx-demo4-1778249282-mtvjc       1/1       Running   3          1d
default       nginx-demo5-2305231528-v0r78       1/1       Running   0          58m
kube-system   etcd-master-1                      1/1       Running   3          1d
kube-system   kube-apiserver-master-1            1/1       Running   3          1d
kube-system   kube-controller-manager-master-1   1/1       Running   3          1d
kube-system   kube-dns-2425271678-kgb7k          3/3       Running   9          1d
kube-system   kube-flannel-ds-049t3              2/2       Running   0          1h
kube-system   kube-flannel-ds-jmpqh              2/2       Running   7          1h
kube-system   kube-flannel-ds-qswt7              2/2       Running   10         1d
kube-system   kube-proxy-6263z                   1/1       Running   5          1d
kube-system   kube-proxy-b8hc3                   1/1       Running   3          1d
kube-system   kube-proxy-vrk60                   1/1       Running   2          1d
kube-system   kube-scheduler-master-1            1/1       Running   3          1d

Get all services: $ kubectl get services

NAME          CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    10.96.0.1       <none>        443/TCP        1d
nginx-demo    10.104.34.229   <none>        80/TCP         1d
nginx-demo2   10.102.145.89   <none>        80/TCP         1d
nginx-demo3   10.103.62.101   <nodes>       80:31844/TCP   1d
nginx-demo4   10.98.4.237     <nodes>       80:30029/TCP   1d
nginx-demo5   10.100.112.76   <nodes>       80:31494/TCP   58m

Get 2 Nginx pods IP info:

$ kubectl get pod nginx-demo4-1778249282-mtvjc -o json | grep IP
        "hostIP": "10.0.13.105",
        "podIP": "10.244.1.80",

$ kubectl get pod nginx-demo5-2305231528-v0r78 -o json | grep IP
        "hostIP": "10.0.13.106",
        "podIP": "10.244.2.21",

As you see - one Nginx example is on the first minion, and the second example is on the second minion.

The problem is - I can access nginx-demo4 from node 10.0.13.105 only (with Pod IP and Service IP), with curl:

curl 10.98.4.237:80
 curl 10.244.1.80:80

, and nginx-demo5 from 10.0.13.106 only, not vice versa and not from master-node. Busybox is on node 10.0.13.105, so it can reach nginx-demo4, but not nginx-demo5.

Routing table on master:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     100    0        0 enp0s3
10.0.13.0       0.0.0.0         255.255.255.0   U     100    0        0 enp0s8
10.244.0.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
10.244.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

Routing table on minion-1:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     100    0        0 enp0s3
10.0.13.0       0.0.0.0         255.255.255.0   U     100    0        0 enp0s8
10.244.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

Decreasing Guest OS security was done:

setenforce 0
systemctl stop firewalld

and bridge config in sysctl:

sysctl -w net.bridge.bridge-nf-call-iptables=1
sysctl -w net.bridge.bridge-nf-call-ip6tables=1

Ports scanning from Host returns:

# nmap 10.0.13.105 -p1-50000

Starting Nmap 7.60 ( https://nmap.org ) at 2017-09-20 14:04 EEST
Nmap scan report for 10.0.13.105
Host is up (0.0020s latency).
Not shown: 49993 closed ports
PORT      STATE    SERVICE
22/tcp    open     ssh
10250/tcp open     unknown
10255/tcp open     unknown
10256/tcp open     unknown
30029/tcp open     unknown
31494/tcp filtered unknown
31844/tcp open     unknown
MAC Address: 08:00:27:F8:E3:71 (Oracle VirtualBox virtual NIC)

Nmap done: 1 IP address (1 host up) scanned in 1.75 seconds

If we take the latest Nginx service on port 31494 - it exists everywhere, but is Open only on related node, on the others it’s filtered.