kubernetes: iptables rules causing network unreachable

*What happened: Pods on 10.244.0.0/16 network can reach each other but cannot reach the 10.96 network or node network.

E0923 21:25:30.881753 1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:317: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8m35s

dnstools# nc -v 10.96.0.1:443 nc: 10.96.0.1:443 (10.96.0.1:443): Network unreachable

[root@server1 ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination KUBE-SERVICES all – anywhere anywhere ctstate NEW /* kubernetes service portals / KUBE-EXTERNAL-SERVICES all – anywhere anywhere ctstate NEW / kubernetes externally-visible service portals */ KUBE-FIREWALL all – anywhere anywhere

Chain FORWARD (policy ACCEPT) target prot opt source destination KUBE-FORWARD all – anywhere anywhere /* kubernetes forwarding rules / KUBE-SERVICES all – anywhere anywhere ctstate NEW / kubernetes service portals */ ACCEPT all – server1.lan/16 anywhere ACCEPT all – anywhere server1.lan/16

Chain OUTPUT (policy ACCEPT) target prot opt source destination KUBE-SERVICES all – anywhere anywhere ctstate NEW /* kubernetes service portals */ KUBE-FIREWALL all – anywhere anywhere

Chain KUBE-EXTERNAL-SERVICES (1 references) target prot opt source destination

Chain KUBE-FIREWALL (2 references) target prot opt source destination DROP all – anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references) target prot opt source destination DROP all – anywhere anywhere ctstate INVALID ACCEPT all – anywhere anywhere /* kubernetes forwarding rules / mark match 0x4000/0x4000 ACCEPT all – server1.lan/16 anywhere / kubernetes forwarding conntrack pod source rule / ctstate RELATED,ESTABLISHED ACCEPT all – anywhere server1.lan/16 / kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (3 references) target prot opt source destination REJECT tcp – anywhere 10.96.0.10 /* kube-system/kube-dns:dns-tcp has no endpoints / tcp dpt:domain reject-with icmp-port-unreachable REJECT tcp – anywhere 10.96.0.10 / kube-system/kube-dns:metrics has no endpoints / tcp dpt:9153 reject-with icmp-port-unreachable REJECT udp – anywhere 10.96.0.10 / kube-system/kube-dns:dns has no endpoints */ udp dpt:domain reject-with icmp-port-unreachable

What you expected to happen:

How to reproduce it (as minimally and precisely as possible): kubadm init --pod-network-cidr=10.244.0.0/16

Anything else we need to know?: Running CRI-O. selinux is enforcing and firewalld is running. Kubernetes 1.13 does not have this issue (our current production version). It appears somethings have changed regarding selinux and iptables from 1.13 to 1.15.

Environment:

• Kubernetes version (use kubectl version): Kubernetes v1.15.3
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release): Red Hat Enterprise Linux Server release 7.7 (Maipo)
• Kernel (e.g. uname -a): inux server1.lan 3.10.0-1062.1.1.el7.x86_64 #1 SMP Tue Aug 13 18:39:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Network plugin and version (if this is a network-related bug): containernetworking-plugins-0.8.1 and Flannel v0.11.0
• Others: crio version 1.15.1-2.el7