kubernetes: Redhat 8 install kubernetes, that can't communication between the podip

What happened: Redhat 8 install kubernetes, that can’t communication between the podip

How to reproduce it (as minimally and precisely as possible): install kubernetes in redhat 8.

# install kubelet、kubeadm、kubectl
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet
systemctl start kubelet

# set firewall
firewall-cmd --zone=public --permanent --add-port={6443,2379,2380,10250,10251,10252}/tcp

# install cluster
kubeadm init   --pod-network-cidr=10.254.0.0/16 --image-repository registry.aliyuncs.com/google_containers

# to start using cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# install calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# use master node
kubectl taint nodes --all node-role.kubernetes.io/master-

podA (busybox): 10.254.219.69 podB (nginx): 10.254.219.68，listen 80 tcp port

kubectl exec -it podA – sh

/ # wget 10.254.219.68:80
Connecting to 10.254.219.68:80 (10.254.219.68:80)
wget: can't connect to remote host (10.254.219.68): No route to host

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:59:43Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:51:19Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):

NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2"

Kernel (e.g. uname -a):

Linux localhost.localdomain 4.18.0-193.el8.x86_64 #1 SMP Fri Mar 27 14:35:58 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools:
Network plugin and version (if this is a network-related bug):

calico

Others: The problem was that the firewalld dropped the packet forwarding packet.

like this:

[root@localhost ~]# nft list chain inet firewalld filter_FORWARD
table inet firewalld {
        chain filter_FORWARD {
                type filter hook forward priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                ip6 daddr { ::/96} reject with icmpv6 type addr-unreachable
                jump filter_FORWARD_IN_ZONES_SOURCE
                jump filter_FORWARD_IN_ZONES
                jump filter_FORWARD_OUT_ZONES_SOURCE
                jump filter_FORWARD_OUT_ZONES
                ct state { invalid } drop
                reject with icmpx type admin-prohibited
        }
}

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 24 (14 by maintainers)

Most upvoted comments

Hi,

I solved this problem in CentOS 8 by creating a new firewalld zone for kubernetes pods and setting its target to ACCEPT. So, firewalld will accept packets going into POD SUBNET CIDR (ingress zone) and also packets coming out of POD SUBNET CIDR (egress zone)

Commands :

firewall-cmd --permanent --delete-zone=kubernetes_pods
firewall-cmd --permanent --new-zone=kubernetes_pods
firewall-cmd --permanent --zone=kubernetes_pods --set-target=ACCEPT
firewall-cmd --permanent --zone=kubernetes_pods --add-source=<POD SUBNET CIDR>
firewall-cmd --reload

firewall-cmd man :

       --permanent [--zone=zone] --set-target=target
           Set the target of a permanent zone.  target is one of: default, ACCEPT, DROP, REJECT

           default is similar to REJECT, but has special meaning in the following scenarios:

            1. ICMP explicitly allowed

               At the end of the zone's ruleset ICMP packets are explicitly allowed.

            2. forwarded packets follow the target of the egress zone

               In the case of forwarded packets, if the ingress zone uses default then whether or not the packet will be
               allowed is determined by the egress zone.

               For a forwarded packet that ingresses zoneA and egresses zoneB:

               ·   if zoneA's target is ACCEPT, DROP, or REJECT then the packet is accepted, dropped, or rejected
                   respectively.

               ·   if zoneA's target is default, then the packet is accepted, dropped, or rejected based on zoneB's target. If
                   zoneB's target is also default, then the packet will be rejected by firewalld's catchall reject.

            3. Zone drifting from source-based zone to interface-based zone

               This only applies if AllowZoneDrifting is enabled. See firewalld.conf(5).

               If a packet ingresses a source-based zone with a target of default, it may still enter an interface-based zone
               (including the default zone).

Versions :

firewall-cmd --version
0.8.2

uname -r
4.18.0-240.10.1.el8_3.x86_64

cat /etc/redhat-release
CentOS Linux release 8.3.2011

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

To see what is getting rejected by firewalld, use the below commands

firewall-cmd --set-log-denied=all
firewall-cmd --reload
dmesg | grep -i reject

sagarvelankar on Feb 28, 2021

kubespray recommend to just stop firewalld, or to manage it manualy I personally leave firewalld enabled with basically the following rules

firewall-cmd --permanent --new-zone=k8s-cluster
firewall-cmd --permanent --new-ipset=k8s-cluster --type=hash:net
firewall-cmd --permanent --zone=k8s-cluster --set-target=ACCEPT
firewall-cmd --permanent --zone=k8s-cluster --add-source=ipset:k8s-cluster
firewall-cmd --permanent --ipset=k8s-cluster --add-entry= --add-entry= ...
firewall-cmd --reload

In the add-entry I put the nodes ips, the clusterIPs and podIPs cidr I have no idea if it affect Calico network policies as I don’t use them, but for sure your issue here is firewalld conf

your nft rule just allow any forwarding, might be too permissive

champtar on Dec 22, 2020

I’m not convinced that this is a documentation issue for Kubernetes. Maybe this’d work better as a Calico issue or support request?

sftim on Dec 22, 2020

This looks like a request for advice. The steps to reproduce: “install kubernetes in redhat 8” might not be detailed enough to replicate the issue, even knowing that Calico was also used. /kind support

tks，I added the installation process and update the comment

@readerx Looks like you solved your issue report. If yes, could you please clarify it was firewalld settings? Could you please share what you did to solve it and close the issue report?

dougsland on Dec 21, 2020