flannel: one pod cannot access clusterIP of another pod that running on the same node

I have two pods on the same node.

openstack-exporter-5555444865-bctlh                1/1     Running     1 (4h55m ago)   6h33m   10.244.2.61      k8s-worker-4.pacific-textiles.com

prometheus-server-6c549c7d4b-fmfvz                 2/2     Running     0               4h22m   10.244.2.98      k8s-worker-4.pacific-textiles.com

However, prometheus-server cannot access ClusterIP of openstack-exporter.

openstack-exporter                      ClusterIP   10.101.122.233   <none>        9180/TCP            6h43m

Here is tcpdump against cni0.

ubuntu@k8s-worker-4:~$ sudo tcpdump -i cni0  -vvv host 10.101.122.233  -n
tcpdump: listening on cni0, link-type EN10MB (Ethernet), capture size 262144 bytes


17:31:08.094903 IP (tos 0x0, ttl 64, id 38276, offset 0, flags [DF], proto TCP (6), length 60)
    10.244.2.98.56726 > 10.101.122.233.9180: Flags [S], cksum 0x92d2 (incorrect -> 0x40c0), seq 816090733, win 62370, options [mss 8910,sackOK,TS val 1403735983 ecr 0,nop,wscale 7], length 0
17:31:09.102061 IP (tos 0x0, ttl 64, id 38277, offset 0, flags [DF], proto TCP (6), length 60)
    10.244.2.98.56726 > 10.101.122.233.9180: Flags [S], cksum 0x92d2 (incorrect -> 0x3cd1), seq 816090733, win 62370, options [mss 8910,sackOK,TS val 1403736990 ecr 0,nop,wscale 7], length 0
17:31:11.118078 IP (tos 0x0, ttl 64, id 38278, offset 0, flags [DF], proto TCP (6), length 60)
    10.244.2.98.56726 > 10.101.122.233.9180: Flags [S], cksum 0x92d2 (incorrect -> 0x34f1), seq 816090733, win 62370, options [mss 8910,sackOK,TS val 1403739006 ecr 0,nop,wscale 7], length 0

We can see clusterIP does not return anything.

I use flannel vXLAN and ipvs mode. This only happens on this node, the other node has no such problem. Please help, thanks.

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (6 by maintainers)

Most upvoted comments

ubuntu@k8s-worker-4:~$ sudo  iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 448 packets, 63483 bytes)
 pkts bytes target     prot opt in     out     source               destination
2954K  353M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain INPUT (policy ACCEPT 2509 packets, 5472K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 262 packets, 32674 bytes)
 pkts bytes target     prot opt in     out     source               destination
1342K  217M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT 2494 packets, 1204K bytes)
 pkts bytes target     prot opt in     out     source               destination
  15M   23G KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
1109K   86M RETURN     all  --  *      *       10.244.0.0/16        10.244.0.0/16        /* flanneld masq */
1060K   60M MASQUERADE  all  --  *      *       10.244.0.0/16       !224.0.0.0/4          /* flanneld masq */ random-fully
    0     0 RETURN     all  --  *      *      !10.244.0.0/16        10.244.2.0/24        /* flanneld masq */
    4   292 MASQUERADE  all  --  *      *      !10.244.0.0/16        10.244.0.0/16        /* flanneld masq */ random-fully

Chain KUBE-KUBELET-CANARY (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-LOAD-BALANCER (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-MARK-DROP (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (3 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODE-PORT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes nodeport TCP port for masquerade purpose */ match-set KUBE-NODE-PORT-TCP dst

Chain KUBE-POSTROUTING (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose */ match-set KUBE-LOOP-BACK dst,dst,src
 2715 1217K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x4000/0x4000
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK xor 0x4000
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ random-fully

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *      !10.244.0.0/16        0.0.0.0/0            /* Kubernetes service cluster ip + port for masquerade purpose */ match-set KUBE-CLUSTER-IP dst,dst
  370 64182 KUBE-NODE-PORT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
  163 14956 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set KUBE-CLUSTER-IP dst,dst

====

ubuntu@k8s-worker-4:~$ sudo iptables -vnL
Chain INPUT (policy ACCEPT 2336 packets, 1167K bytes)
 pkts bytes target     prot opt in     out     source               destination
  45M  127G KUBE-PROXY-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-proxy firewall rules */
  45M  127G KUBE-NODE-PORT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes health check rules */
  45M  127G KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  54M  103G KUBE-PROXY-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-proxy firewall rules */
  54M  103G KUBE-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
1253K   72M ACCEPT     all  --  *      *       10.244.0.0/16        0.0.0.0/0            /* flanneld forward */
   78  164K ACCEPT     all  --  *      *       0.0.0.0/0            10.244.0.0/16        /* flanneld forward */

Chain OUTPUT (policy ACCEPT 2019 packets, 3514K bytes)
 pkts bytes target     prot opt in     out     source               destination
  41M  148G KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-FIREWALL (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *      !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
 3964 3609K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-NODE-PORT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kubernetes health check node port */ match-set KUBE-HEALTH-CHECK-NODE-PORT dst

Chain KUBE-PROXY-FIREWALL (2 references)
 pkts bytes target     prot opt in     out     source               destination

Chain KUBE-SOURCE-RANGES-FIREWALL (0 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

luckydogxf on Mar 1, 2023