calico: Pod communication broken after upgrade from Canal v3.19.1 to v3.22.0

After upgrade of Canal from v3.19.1 to v3.22.0, the pod communication is broken for newly started pods. It is working only for pods that were running before the upgrade. The issue can be recovered by reboot of the node.

Expected Behavior

After CNI upgrade, the newly started pods should be able to communicate with any other pod.

Current Behavior

A pod started after the upgrade is not able to communicate with any other pod in the cluster.

For example, the pod A with IP 10.244.3.9 is trying to ping the pod B with IP 10.244.3.10. iptables trace on the host shows that the packet is dropped by the cali-from-wl-dispatch rule:

trace id 76f540ab ip raw PREROUTING packet: iif "calie5da25cfd1a" ether saddr 76:0e:b8:ba:fa:08 ether daddr ee:ee:ee:ee:ee:ee ip saddr 10.244.3.9 ip daddr 10.244.3.10 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 60250 ip length 84 icmp type echo-request icmp code net-unreachable icmp id 58087 icmp sequence 1 @th,64,96 857356887478869055311578368
trace id 76f540ab ip raw PREROUTING rule meta l4proto icmp ip daddr 10.244.3.10 counter packets 11 bytes 924 meta nftrace set 1 (verdict continue)
trace id 76f540ab ip raw PREROUTING verdict continue meta mark 0x00040000
trace id 76f540ab ip raw PREROUTING policy accept meta mark 0x00040000
trace id 76f540ab ip mangle PREROUTING packet: iif "calie5da25cfd1a" ether saddr 76:0e:b8:ba:fa:08 ether daddr ee:ee:ee:ee:ee:ee ip saddr 10.244.3.9 ip daddr 10.244.3.10 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 60250 ip length 84 icmp type echo-request icmp code net-unreachable icmp id 58087 icmp sequence 1 @th,64,96 857356887478869055311578368
trace id 76f540ab ip mangle PREROUTING rule # xt_comment counter packets 528464 bytes 651457512 jump cali-PREROUTING (verdict jump cali-PREROUTING)
trace id 76f540ab ip mangle cali-PREROUTING rule # xt_comment counter packets 221165 bytes 17736879 jump cali-from-host-endpoint (verdict jump cali-from-host-endpoint)
trace id 76f540ab ip mangle cali-from-host-endpoint verdict continue meta mark 0x00040000
trace id 76f540ab ip mangle cali-PREROUTING verdict continue meta mark 0x00040000
trace id 76f540ab ip mangle PREROUTING verdict continue meta mark 0x00040000
trace id 76f540ab ip mangle PREROUTING policy accept meta mark 0x00040000
trace id 76f540ab ip nat PREROUTING packet: iif "calie5da25cfd1a" ether saddr 76:0e:b8:ba:fa:08 ether daddr ee:ee:ee:ee:ee:ee ip saddr 10.244.3.9 ip daddr 10.244.3.10 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 60250 ip length 84 icmp type echo-request icmp code net-unreachable icmp id 58087 icmp sequence 1 @th,64,96 857356887478869055311578368
trace id 76f540ab ip nat PREROUTING rule # xt_comment counter packets 8305 bytes 691157 jump cali-PREROUTING (verdict jump cali-PREROUTING)
trace id 76f540ab ip nat cali-PREROUTING rule # xt_comment counter packets 8305 bytes 691157 jump cali-fip-dnat (verdict jump cali-fip-dnat)
trace id 76f540ab ip nat cali-fip-dnat verdict continue meta mark 0x00040000
trace id 76f540ab ip nat cali-PREROUTING verdict continue meta mark 0x00040000
trace id 76f540ab ip nat PREROUTING rule # xt_comment counter packets 8305 bytes 691157 jump KUBE-SERVICES (verdict jump KUBE-SERVICES)
trace id 76f540ab ip nat KUBE-SERVICES verdict continue meta mark 0x00040000
trace id 76f540ab ip nat PREROUTING verdict continue meta mark 0x00040000
trace id 76f540ab ip nat PREROUTING policy accept meta mark 0x00040000
trace id 76f540ab ip mangle FORWARD packet: iif "calie5da25cfd1a" oif "cali8efcee298dd" ether saddr 76:0e:b8:ba:fa:08 ether daddr ee:ee:ee:ee:ee:ee ip saddr 10.244.3.9 ip daddr 10.244.3.10 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 60250 ip length 84 icmp type echo-request icmp code net-unreachable icmp id 58087 icmp sequence 1 @th,64,96 857356887478869055311578368
trace id 76f540ab ip mangle FORWARD verdict continue meta mark 0x00040000
trace id 76f540ab ip mangle FORWARD policy accept meta mark 0x00040000
trace id 76f540ab ip filter FORWARD packet: iif "calie5da25cfd1a" oif "cali8efcee298dd" ether saddr 76:0e:b8:ba:fa:08 ether daddr ee:ee:ee:ee:ee:ee ip saddr 10.244.3.9 ip daddr 10.244.3.10 ip dscp cs0 ip ecn not-ect ip ttl 63 ip id 60250 ip length 84 icmp type echo-request icmp code net-unreachable icmp id 58087 icmp sequence 1 @th,64,96 857356887478869055311578368
trace id 76f540ab ip filter FORWARD rule # xt_comment counter packets 11739 bytes 3728960 jump cali-FORWARD (verdict jump cali-FORWARD)
trace id 76f540ab ip filter cali-FORWARD rule # xt_comment counter packets 11739 bytes 3728960 # xt_MARK (verdict continue)
trace id 76f540ab ip filter cali-FORWARD rule iifname "cali*" # xt_comment counter packets 9905 bytes 813300 jump cali-from-wl-dispatch (verdict jump cali-from-wl-dispatch)
trace id 76f540ab ip filter cali-from-wl-dispatch rule # xt_comment # xt_comment counter packets 8053 bytes 676400 drop (verdict drop)

It is dropped by the last rule in the cali-from-wl-dispatch chain:

Chain cali-from-wl-dispatch (2 references)
 pkts bytes target     prot opt in     out     source               destination
 1634  135K cali-fw-cali0eacf697eec  all  --  cali0eacf697eec *       0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:jfxMCpNf8Nj1KM-K */
 1666  138K cali-fw-cali9b778b31de0  all  --  cali9b778b31de0 *       0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:Z5r7QQDW3XChII0Z */
 1529  111K cali-fw-caliec620394ae2  all  --  caliec620394ae2 *       0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:j5kopEPZZL4XyLX1 */
 8054  676K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:jP5qeZOJds6jo3_6 */ /* Unknown interface */

the chain cali-from-wl-dispatch does not seem to contain rules for the involved interface names, but contains rules for non-existing interface names:

$ ip route | grep cali
10.244.3.5 dev calib3c61c3cba9 scope link
10.244.3.9 dev calie5da25cfd1a scope link
10.244.3.10 dev cali8efcee298dd scope link

$ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:fc:08:ef brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 192.168.1.250/24 metric 1024 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 67408sec preferred_lft 67408sec
    inet6 fe80::f816:3eff:fefc:8ef/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:7b:2a:96:2d brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether d2:61:31:39:5c:29 brd ff:ff:ff:ff:ff:ff
    inet 10.244.3.0/32 brd 10.244.3.0 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::d061:31ff:fe39:5c29/64 scope link
       valid_lft forever preferred_lft forever
13: calib3c61c3cba9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-870bafee-f33a-9ff8-78d4-6ded80ae5f55
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
16: nodelocaldns: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether ba:d3:80:2a:6c:ae brd ff:ff:ff:ff:ff:ff
    inet 169.254.20.10/32 brd 169.254.20.10 scope global nodelocaldns
       valid_lft forever preferred_lft forever
    inet6 fe80::b8d3:80ff:fe2a:6cae/64 scope link
       valid_lft forever preferred_lft forever
20: calie5da25cfd1a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-507e2e66-63e7-5601-4b53-d29aaa78e282
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
21: cali8efcee298dd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-2ff23a01-36db-5e18-4caf-c01ebc59454f
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever

iptables dump: iptables-dump.txt

There is no error in the calico-node logs, only a few INFO logs that might be related:

2022-03-15 12:13:00.734 [INFO][47] felix/route_table.go 1116: Failed to access interface because it doesn't exist. error=Link not found ifaceName="cali0eacf697eec" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0
2022-03-15 12:13:00.734 [INFO][47] felix/route_table.go 1184: Failed to get interface; it's down/gone. error=Link not found ifaceName="cali0eacf697eec" ifaceRegex="^cali.*" ipVersion=0x4 tableIndex=0

calico-node logs: calico-node-logs.txt kube-flannel logs: kube-flannel-logs.txt

Steps to Reproduce (for bugs)

1. Deploy a cluster with Canal v3.19.1
2. Start a bunch of pods
3. Upgrade Canal to to v3.22.0
4. Start a new pod
5. The new pod is not able to communicate with any other pods

Your Environment

• Calico version v3.22.0
• Orchestrator version (e.g. kubernetes, mesos, rkt): k8s v1.21.6
• Operating System and version: Flatcar Container Linux stable 3033.2.3 (kernel 5.10.102-flatcar)