kubernetes: Kubernetes, can't reach other node services

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened:

I’m playing with Kubernetes/kubeadm inside 3 VirtualBox VMs with CentOS 7, 1 master and 2 minions. Unfortunately installation manuals say something like every service will be accessible from every node, every pod will see all other pods, but I don’t see this happening. I can access the service only from that node where the specific pod runs.

What you expected to happen:

Every service is accessible from every node, every pod can see all other pods.

How to reproduce it (as minimally and precisely as possible):

Use kubeadm to setup 1 master and 2 minions, all with CentOS 7, like here is written: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/, Flannel for networking, I just replaced demo app with simple Nginx pod. Expose the service with default type or NodePort and try to reach it from other VMs and from Host.

Environment:

  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-14T06:36:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration**:
  • OS: CentOS Linux 7 (Core)
  • Kernel: 3.10.0-693.2.2.el7.x86_64
  • Install tools: kubeadm
  • VirtualBox: 5.1.26 r117224

Anything else we need to know?:

Every VM has 2 adapters: NAT and Host-only. Second one has IPs 10.0.13.101-254.

  • Master-1: 10.0.13.104
  • Minion-1: 10.0.13.105
  • Minion-2: 10.0.13.106

Get all pods from master:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY     STATUS    RESTARTS   AGE
default       busybox                            1/1       Running   12         1d
default       nginx-demo-2867147694-f6f9m        1/1       Running   4          1d
default       nginx-demo2-2631277934-hd7jz       1/1       Running   2          1d
default       nginx-demo3-2046388352-6zj6t       1/1       Running   2          1d
default       nginx-demo4-1778249282-mtvjc       1/1       Running   3          1d
default       nginx-demo5-2305231528-v0r78       1/1       Running   0          58m
kube-system   etcd-master-1                      1/1       Running   3          1d
kube-system   kube-apiserver-master-1            1/1       Running   3          1d
kube-system   kube-controller-manager-master-1   1/1       Running   3          1d
kube-system   kube-dns-2425271678-kgb7k          3/3       Running   9          1d
kube-system   kube-flannel-ds-049t3              2/2       Running   0          1h
kube-system   kube-flannel-ds-jmpqh              2/2       Running   7          1h
kube-system   kube-flannel-ds-qswt7              2/2       Running   10         1d
kube-system   kube-proxy-6263z                   1/1       Running   5          1d
kube-system   kube-proxy-b8hc3                   1/1       Running   3          1d
kube-system   kube-proxy-vrk60                   1/1       Running   2          1d
kube-system   kube-scheduler-master-1            1/1       Running   3          1d

Get all services: $ kubectl get services

NAME          CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes    10.96.0.1       <none>        443/TCP        1d
nginx-demo    10.104.34.229   <none>        80/TCP         1d
nginx-demo2   10.102.145.89   <none>        80/TCP         1d
nginx-demo3   10.103.62.101   <nodes>       80:31844/TCP   1d
nginx-demo4   10.98.4.237     <nodes>       80:30029/TCP   1d
nginx-demo5   10.100.112.76   <nodes>       80:31494/TCP   58m

Get 2 Nginx pods IP info:

$ kubectl get pod nginx-demo4-1778249282-mtvjc -o json | grep IP
        "hostIP": "10.0.13.105",
        "podIP": "10.244.1.80",

$ kubectl get pod nginx-demo5-2305231528-v0r78 -o json | grep IP
        "hostIP": "10.0.13.106",
        "podIP": "10.244.2.21",

As you see - one Nginx example is on the first minion, and the second example is on the second minion.

The problem is - I can access nginx-demo4 from node 10.0.13.105 only (with Pod IP and Service IP), with curl:

curl 10.98.4.237:80
 curl 10.244.1.80:80

, and nginx-demo5 from 10.0.13.106 only, not vice versa and not from master-node. Busybox is on node 10.0.13.105, so it can reach nginx-demo4, but not nginx-demo5.

Routing table on master:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     100    0        0 enp0s3
10.0.13.0       0.0.0.0         255.255.255.0   U     100    0        0 enp0s8
10.244.0.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
10.244.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

Routing table on minion-1:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     100    0        0 enp0s3
10.0.13.0       0.0.0.0         255.255.255.0   U     100    0        0 enp0s8
10.244.0.0      0.0.0.0         255.255.0.0     U     0      0        0 flannel.1
10.244.1.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0

Decreasing Guest OS security was done:

setenforce 0
systemctl stop firewalld

and bridge config in sysctl:

sysctl -w net.bridge.bridge-nf-call-iptables=1
sysctl -w net.bridge.bridge-nf-call-ip6tables=1

Ports scanning from Host returns:

# nmap 10.0.13.105 -p1-50000

Starting Nmap 7.60 ( https://nmap.org ) at 2017-09-20 14:04 EEST
Nmap scan report for 10.0.13.105
Host is up (0.0020s latency).
Not shown: 49993 closed ports
PORT      STATE    SERVICE
22/tcp    open     ssh
10250/tcp open     unknown
10255/tcp open     unknown
10256/tcp open     unknown
30029/tcp open     unknown
31494/tcp filtered unknown
31844/tcp open     unknown
MAC Address: 08:00:27:F8:E3:71 (Oracle VirtualBox virtual NIC)

Nmap done: 1 IP address (1 host up) scanned in 1.75 seconds

If we take the latest Nginx service on port 31494 - it exists everywhere, but is Open only on related node, on the others it’s filtered.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 3
  • Comments: 20 (4 by maintainers)

Most upvoted comments

I had exactly the same issue on my cfcr-deployment on openstack. nodeport was only published and accessible on the worker where the pod was deployed. The nodeport on the remaining workers were ‘filtered’. I simply solved it by opening udp ports on the underlying openstack-securitygroups. This helped (as far as I understood) because exposing nodeports to other workers uses vxlan which uses udp. without udp no connection can be done and e.g. nmap shows “filtered” ports.

hope this might help some of you guys!

cheers

@camflan I am struggling with the same problem. Would you be so kind of giving more details about how to solve it?

I am also using Flannel, setting --pod-network-cidr=10.244.0.0/16 on the master node with kuberadm init and initializing the network with kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

The whole description of my problem is here: https://stackoverflow.com/questions/50401355/requests-timing-out-when-accesing-a-kubernetes-clusterip-service