minikube: Minikube to host communication not working on Fedora 37

What Happened?

There seems to be a difference in minikube iptables rules when comparing a fully updated Fedora 36 and Fedora 37 system. On Fedora 36:

$ sudo iptables -t nat -S|grep -e '--to-destination 127.0.0.11'
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:39397
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:34196

On Fedora 37:

$ sudo iptables -t nat -S|grep -e '--to-destination 127.0.0.11'
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p tcp -j DNAT --to-destination 127.0.0.11:46739
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p udp -j DNAT --to-destination 127.0.0.11:37392

The missing --dport 53 condition on the destination NAT breaks all non-DNS communication between host and minikube. What might be causing this difference?

Attach the log file

log.txt

Operating System

Redhat/Fedora

Driver

Docker

About this issue

Original URL
State: open
Created a year ago
Comments: 19 (9 by maintainers)

Most upvoted comments

I guess that means that the docker rules are fine until minikube starts patching them with iptables-save | iptables-restore? Would it be possible for minikube/kind to just remove the docker rules and then create the needed rules from scratch?

So KIND doesn’t use save+restore itself, but kube-proxy in Kubernetes works this way for good reasons (reconciling a large set of rules) and there’s a third version of iptables in the kube-proxy image. I suspect the same for minikube.

On your host with minikube it’s 1.8.8 nf_tables, 1.8.4 ?, and then 1.8.7 ? (kube-proxy in 1.25). In KIND it will be 1.8.8 nf_tables (host), 1.8.7 nf_tables (node), 1.8.7 nf_tables (kube-proxy). There’s actually a fourth for CNI … but that’s generally kube-proxy matching more or less.

Discussing with @aojea and https://github.com/kubernetes-sigs/kind/pull/3059 how we might work around this.

One thought is multi-version selecting the binaries and attempting to detect what the host is using.

Another is the workaround in https://github.com/kubernetes-sigs/kind/pull/3059 combined with making sure at least kube-proxy + kindnetd + kind node match. Which is a trickier proposition for additional CNIs minikube may support.

BenTheElder on Jan 9, 2023

This works fine (on my VM) 😄 https://github.com/kubernetes-sigs/kind/pull/3059

aojea on Jan 8, 2023

Yes, as @BenTheElder mentions, this indeed seems to be another case of incompatibility between 1.8.8 and 1.8.7. Without involving minikube, the problem can bee seen by:

$ docker network create --driver bridge test-net
$ docker run -it --privileged --network test-net fedora:37 bash
$ dnf install iptables-nft nftables
$ iptables-nft-save 
# Generated by iptables-nft-save v1.8.8 (nf_tables) on Sat Jan  7 17:44:56 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER_OUTPUT - [0:0]
:DOCKER_POSTROUTING - [0:0]
-A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 127.0.0.11/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:41759
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:43231
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -m tcp --sport 41759 -j SNAT --to-source :53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 43231 -j SNAT --to-source :53
COMMIT
$ exit
$ docker run -it --privileged --network test-net fedora:36 bash
$ dnf install iptables-nft nftables
$ iptables-nft-save 
# Generated by iptables-nft-save v1.8.7 on Sat Jan  7 18:31:05 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER_OUTPUT - [0:0]
:DOCKER_POSTROUTING - [0:0]
-A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 127.0.0.11/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -j DNAT --to-destination 127.0.0.11:38683
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -j DNAT --to-destination 127.0.0.11:57275
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -j SNAT --to-source :53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -j SNAT --to-source :53
COMMIT

Both 1.8.8 and 1.8.7 does however produce similar output from nft list ruleset:

table ip nat {
	chain DOCKER_OUTPUT {
		ip daddr 127.0.0.11 tcp dport 53 counter packets 0 bytes 0 dnat to 127.0.0.11:38683
		ip daddr 127.0.0.11 udp dport 53 counter packets 35 bytes 2730 dnat to 127.0.0.11:57275
	}
...

I guess that means that the docker rules are fine until minikube starts patching them with iptables-save | iptables-restore?

Would it be possible for minikube/kind to just remove the docker rules and then create the needed rules from scratch?

mnk on Jan 8, 2023

thanks for sharing the details @BenTheElder ! that’s an interesting conversation between Dan (kubernetes) and Phil (netfilter) so, iptables-nft v1.8.8 introduced a breaking change, and there are no plans to “fix” that, and the workaround atm is to:

stick with v1.8.7 (that’s also used in kube-proxy) - both nft and legacy mode should work, or
use iptables (even v1.8.8) in legacy mode (that some linux distros still keep as default)

looks like the full transition from legacy to nft is going to be fun and not so quick

prezha on Jan 7, 2023

@prezha , I tried the minikube build you linked to, but I still get the same result - no --dport 53 condition on the DNAT rule. Do I need to specify a base image or just do a minikube-linux-amd64 start?

Do you get the --dport 53 condition when testing with you branch?

My use-case can be tested by starting sshd on the host and then doing minikube ssh ssh $(id -un)@192.168.49.1. This works fine on Fedora 36 but not on Fedora 37.

mnk on Jan 6, 2023

hmm, do we know if was docker the one to drop the port?

I’ve seen a recent report in kubernetes about some weird iptables rules being mutated in Centos

aojea on Jan 4, 2023