k3s: ServiceLB cannot be accessed via loopback when service ExternalTrafficPolicy=Local

K3s Version: Migrating from v1.25.6+k3s1 to v1.25.7+k3s1

Node(s) CPU architecture, OS, and Version: 1 Node, x86, reproducible on different OS like Ubuntu 22.04 and Ubuntu on WSL

Cluster Configuration: 1 Node local dev development

Describe the bug: We’re using k3s as our local development environment platform, routing FQDNs to our dev machines using /etc/hosts entries pointing to 127.0.0.1

This worked perfectly fine for basically years now, until recently. I had to dig quite a while before I was able to pinpoint it to upgrading k3s from v1.25.6+k3s1 to v1.25.7+k3s1. It stops working on 1.25.7 and works again after downgrading to 1.25.6. The problem also exists on the most current 1.27.1

Steps To Reproduce: Install k3s and ingress controller like so and access it using 127.0.0.1

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.25.7+k3s1 K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="--disable=traefik" sh -

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install -n kube-system --version '^4.0.0' --set 'controller.watchIngressWithoutClass=true' --set 'controller.service.externalTrafficPolicy=Local' nginx-ingress ingress-nginx/ingress-nginx

curl http://127.0.0.1

Expected behavior: A 404 Message returned by nginx

Actual behavior: Network timeout

However - the Ports/Services are in fact working perfectly fine using the Node’s LAN Address like 192.168… It’s just not available via 127.0.0.1 anymore and I can’t figure out, why.

Additional info

• I checked the corresponding changelog at https://github.com/k3s-io/k3s/releases/tag/v1.25.7%2Bk3s1 and found some changes regarding the servicelb but at least to me none of them explained the behavior I’m seeing.

• I exported iptables while running 1.25.6 and while running 1.25.7 and tried to compare them somehow but I’m afraid my knowledge of iptables is not sufficient to assess whether the cause of the problem is to be found here or not.

Thanks a lot for any help in advance - Max