kubernetes: kube-proxy v1.26 blocks host services

What happened?

We run some Kubernetes “single-node clusters” that combine the control-plane and the workload onto a single host besides some services installed directly on the operating system (SSH, BIND9). With the upgrade to v1.26, all local services become unreachable as soon as kube-proxy configures the network stack. This happens because kube-proxy adds the node’s IP address to the ipset KUBE-IPVS-IPS.

We configure the node IP address explicitly for kubelet:

KUBELET_EXTRA_ARGS=--container-runtime-endpoint=unix:///run/cri-dockerd.sock --node-ip=10.169.8.6

Also, kube-proxy is configured to exclude the node IP address explicitly:

    ipvs:
      excludeCIDRs:
      - 10.169.8.6/32

@liaralabs has collected more information in https://github.com/k3s-io/k3s/issues/7183#issuecomment-1491205024.

Overriding the iptables rules from kube-proxy with a higher-priority rules makes the services reachable again. Also, removing the node’s IP address from the ipset helps (until kube-proxy syncs the set again).

What did you expect to happen?

kube-proxy should keep away from node-local services.

How can we reproduce it (as minimally and precisely as possible)?

• Setup a minimal one-node cluster with kubeadm and configure it for ipvs.
• Add a Service of type LoadBalancer with externalIPs configured.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:13:53Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:05:35Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux bbs 5.19.0-40-generic #41~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 31 16:00:14 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)