k3s: Networking/NAT not working on older iptables versions with v1.25.7+k3s1

Environmental Info: K3s Version: k3s version v1.25.7+k3s1 (f7c20e23) go version go1.19.6

**Node(s) CPU architecture, OS, and Version: **

Linux dev01-spa 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux Running CentOS Linux release 7.9.2009

Cluster Configuration: Single node.

Describe the bug: Pods can not access external networks. This works fine if going back to v1.25.6+k3s1 It turns out outgoing traffic from the host is not NATed anymore.

e.g. running tcpdump on the ens32 interface, which is the NIC owning the IP 192.168.170.101 of the host, that’s attached to the local subnet.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens32, link-type EN10MB (Ethernet), capture size 262144 bytes
09:47:47.668554 ARP, Request who-has 192.168.170.101 (00:50:56:bd:82:75) tell 192.168.170.172, length 46
09:47:47.668566 ARP, Reply 192.168.170.101 is-at 00:50:56:bd:82:75, length 28
09:47:48.018543 IP 192.168.170.101.41440 > 192.168.170.226.9092: Flags [.], ack 17, win 15708, options [nop,nop,TS val 568150586 ecr 3186318162], length 0
09:47:48.069123 IP 10.42.0.18 > 8.8.8.8: ICMP echo request, id 7, seq 3, length 64
09:47:48.145339 STP 802.1w, Rapid STP, Flags [Proposal, Learn, Forward], bridge-id 8000.ec:02:73:2e:91:40.8017, length 43
09:47:49.069190 IP 10.42.0.18 > 8.8.8.8: ICMP echo request, id 7, seq 4, length 64

The line above with `10.42.0.18 > 8.8.8.8:? is the result of a pod manually doing a ping 8.8.8.8 The pod network seems not to be NAT’ed out, the 10.42.0.18 address should not appear on our internal network.

Steps To Reproduce: Installed k3s with with curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik" sh - on CentOS Linux release 7.9.2009

Ran

  kubectl run -it --rm --restart=Never --image busybox tempbusybox -- sh

And manually ran ping 8.8.8.8 from this pod, which never get any replies (see the above tcpdump output)

Expected behavior: With v1.25.6+k3s1 everything works ok on this node.

Installed with curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.25.6+k3s1 INSTALL_K3S_EXEC="--disable traefik" sh - Ran

  kubectl run -it --rm --restart=Never --image busybox tempbusybox -- sh

And manually running ping 8.8.8.8 from the bosybox pod succeeds.

[root@dev01-spa ~]# tcpdump not port 22  -nnn -i ens32
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens32, link-type EN10MB (Ethernet), capture size 262144 bytes
10:01:03.070946 IP 192.168.170.164.5353 > 224.0.0.251.5353: 0 [4q] [1au] PTR (QU)? _hap._tcp.local. PTR (QU)? _hap._udp.local. PTR (QU)? _companion-link._tcp.local. PTR (QU)? _sleep-proxy._udp.local. (119)
10:01:03.070994 IP6 fe80::1474:5c2b:f047:36a2.5353 > ff02::fb.5353: 0 [4q] [1au] PTR (QU)? _hap._tcp.local. PTR (QU)? _hap._udp.local. PTR (QU)? _companion-link._tcp.local. PTR (QU)? _sleep-proxy._udp.local. (119)
10:01:03.109346 IP 192.168.170.101 > 8.8.8.8: ICMP echo request, id 7, seq 0, length 64
10:01:03.122974 IP 8.8.8.8 > 192.168.170.101: ICMP echo reply, id 7, seq 0, length 64

Now the external traffic gets NAT’ed, as seen with 192.168.170.101 > 8.8.8.8: above, 10.42.0…0/24 network is not leaking out of the host.

Additional context / logs: