flannel: cant connect to pods from external k8s nodes , k8s nodes can connect only to local pods, load balancing doesnt work properly

Your Environment

3 servers with CentOS 7.3.1611 (Core) / 3.10.0-514.el7.x86_64 flannel - v0.8.0 kubernetes - v 1.7.5 3 node cluster with 1 master (cent501) and 2 workers (cent502, cent503)

[root@cent501 ~]# kubectl get nodes
NAME      STATUS    AGE       VERSION
cent501   Ready     39m       v1.7.5
cent502   Ready     24m       v1.7.5
cent503   Ready     14m       v1.7.5

Expected Behavior

1.k8s nodes should serve my service (type=nodePort) on servicePort on each k8s node 2.pods should communicate with others, nodes should communicate with pods

Current Behavior

run simple deployment based on 6 replicas of nginx:

[root@cent501 ~]# kubectl run deployment01 --image=nginx --replicas=6 --port=80
deployment "deployment01" created

my pods:

[root@cent501 ~]# kubectl get pods  -o wide
NAME                           READY     STATUS        RESTARTS   AGE       IP            NODE
deployment01-554722758-7lfdg   1/1       Running       0          48s       10.244.1.29   cent502
deployment01-554722758-9gvp5   1/1       Running       0          48s       10.244.2.13   cent503
deployment01-554722758-hdgw2   1/1       Running       0          48s       10.244.0.10   cent501
deployment01-554722758-p5hzs   1/1       Running       0          48s       10.244.2.12   cent503
deployment01-554722758-sxqr2   1/1       Running       0          48s       10.244.2.10   cent503
deployment01-554722758-vxp7j   1/1       Running       0          48s       10.244.2.11   cent503

expose deployment as a service with type “NodePort”

[root@cent501 ~]# kubectl expose deployment/deployment01 --type="NodePort" --port 80
service "deployment01" exposed

my services and my endpoints (nginx service works on 31326):

[root@cent501 ~]# kubectl get svc
NAME           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
deployment01   10.110.20.86   <nodes>       80:31326/TCP   4s
kubernetes     10.96.0.1      <none>        443/TCP        44m

[root@cent501 ~]# kubectl get ep
NAME           ENDPOINTS                                                  AGE
deployment01   10.244.0.10:80,10.244.1.29:80,10.244.2.10:80 + 3 more...   6s
kubernetes     192.168.43.148:6443                                        44m

[root@cent501 tmp]# kubectl describe svc deployment01
Name:			deployment01
Namespace:		default
Labels:			run=deployment01
Annotations:		<none>
Selector:		run=deployment01
Type:			NodePort
IP:			10.110.20.86
Port:			<unset>	80/TCP
NodePort:		<unset>	31326/TCP
Endpoints:		10.244.0.10:80,10.244.1.29:80,10.244.2.10:80 + 3 more...
Session Affinity:	None
Events:			<none>

my flannel subnet.env files on 3 nodes:

# ssh cent501 cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

# ssh cent502 cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

# ssh cent503 cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.2.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

I can’t curl (on service port = 31326) my hardware-nodes, load balancing doesnt work properly I can only curl and ping pods from nodes where these pods are running , no way to communicate with pods from other host

Once again my pods:

[root@cent501 ~]# kubectl get pods  -o wide
NAME                           READY     STATUS    RESTARTS   AGE       IP            NODE
deployment01-554722758-7lfdg   1/1       Running   0          14m       10.244.1.29   cent502
deployment01-554722758-9gvp5   1/1       Running   0          14m       10.244.2.13   cent503
deployment01-554722758-hdgw2   1/1       Running   0          14m       10.244.0.10   cent501
deployment01-554722758-p5hzs   1/1       Running   0          14m       10.244.2.12   cent503
deployment01-554722758-sxqr2   1/1       Running   0          14m       10.244.2.10   cent503
deployment01-554722758-vxp7j   1/1       Running   0          14m       10.244.2.11   cent503

on node cent501 I can only ping these one pod with 10.244.0.10
on node cent502 I can only ping these one pod with 10.244.1.29
on node cent503 I have 4 pods so I can ping them but only them: 10.244.2.13
10.244.2.12
10.244.2.10
10.244.2.11

with curling 80 the same effect

my routing tables:

# ssh cent501 ip route show
default via 192.168.43.1 dev enp0s3  proto static  metric 100 
10.244.0.0/24 dev cni0  proto kernel  scope link  src 10.244.0.1 
10.244.0.0/16 dev flannel.1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.43.0/24 dev enp0s3  proto kernel  scope link  src 192.168.43.148  metric 100 

# ssh cent502 ip route show
default via 192.168.43.1 dev enp0s3  proto static  metric 100 
10.244.0.0/16 dev flannel.1 
10.244.1.0/24 dev cni0  proto kernel  scope link  src 10.244.1.1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.43.0/24 dev enp0s3  proto kernel  scope link  src 192.168.43.116  metric 100 

# ssh cent503 ip route show
default via 192.168.43.1 dev enp0s3  proto static  metric 100 
10.244.0.0/16 dev flannel.1 
10.244.2.0/24 dev cni0  proto kernel  scope link  src 10.244.2.1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.43.0/24 dev enp0s3  proto kernel  scope link  src 192.168.43.163  metric 100