WSL: Missing `xt_recent` kernel module resulting in inability to run Kubernetes with Session Affinity
Windows Build Number
Microsoft Windows [Version 10.0.19042.1055]
WSL Version
- WSL 2
- WSL 1
Kernel Version
No response
Distro Version
Ubuntu 20.04
Other Software
Kind, v0.11.1 Kubernetes, v1.20.7 Docker Desktop (Windows), version 3.4.0
Repro Steps
- Install Docker Desktop (link)
- Get Kind (link)
go get sigs.k8s.io/kind@v0.11.1 - Create a Kubernetes cluster
kind create cluster - Create a Service in Kubernetes with
sessionAffinity: ClientIP(kubectl apply -f service.yaml).# service.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: session-affinity-test spec: selector: matchLabels: app: session-affinity-test template: metadata: labels: app: session-affinity-test spec: containers: - name: session-affinity-test image: mendhak/http-https-echo resources: limits: memory: "128Mi" cpu: "500m" ports: - containerPort: 8080 name: web --- apiVersion: v1 kind: Service metadata: name: session-affinity-service namespace: default spec: ports: - name: web port: 9093 targetPort: web selector: app: session-affinity-test sessionAffinity: ClientIP - Try to access the service, or view Kube Proxy logs
Expected Behavior
Kube Proxy should have successfully applied iptables rules and Kubernetes service is accessible.
Some discussion about this issue: https://github.com/kubernetes-sigs/kind/issues/1740.
Yes, it looks like the current WSL2 Kernel is built without xt_recent, needed by iptables -m recent … which kube-proxy uses to implement sessionAffinity: ClientIP. Custom Kernel built with CONFIG_NETFILTER_XT_MATCH_RECENT=y fixed it for me. Submitted https://github.com/microsoft/WSL2-Linux-Kernel/pull/198 (4.19.y) and https://github.com/microsoft/WSL2-Linux-Kernel/pull/199 (5.4.y)
Actual Behavior
Service is inaccessible and any other service created afterwards is inaccessible. Which is caused by inability to apply iptables rules.
The following log is present in Kube Proxy:
E0720 14:29:10.934607 1 proxier.go:1507] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.8.3 (legacy): Couldn't load match `recent':No such file or directory
Error occurred at line: 96
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
)
I0720 14:29:10.934636 1 proxier.go:779] Sync failed; retrying in 30s
Diagnostic Logs
No response
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 19
- Comments: 18 (1 by maintainers)
this issue has caused me nightmares since the whole cluster wasnt able to communicate anymore just because one container has sessionAffinity: ClientIp set.
@craigloewen-msft Any news on this? It’s been half a year. PRs that addressed this issue were opened over at microsoft/WSL2-Linux-Kernel and closed without a good reason.
@jiayali-ms you closed both PRs to the kernel trying to implement this fix (https://github.com/microsoft/WSL2-Linux-Kernel/pull/199 and https://github.com/microsoft/WSL2-Linux-Kernel/pull/198). On your closing comment, you said:
We are here, reporting the issue that causes so many k8s environments to be broken in WSL 2 and yet we didn’t hear anything back.
So, what’s up? Is this going to be fixed?
Recompiling the WSL 2 kernel only for this reason is an awful solution.
Oh, men, I am trying to provision an RKE2 worker node using Calico on Win11 WSL2 with a bridged network. And the kube-proxy pod suffers from this no xt_recent iptables failure. I am already using a custom kernel (built using native Ubuntu 22.04 and a modified ms kconfig) with many xt options enabled to support wireguard. I wonder why MS is so hostile to the Linux net stack… All they are trying to do is to limit the role of wsl a client behind NAT… besides, I still have several machines not compatible with win11, and need to figure out how to overcome the hyper-v switch nat mode…
MS team - without this fix there’s a whole class of WSL2 use-cases (developer local K8s on WSL2) that are invalidated. Please add this feature in an upcoming build - WSL2 has great potential as a development environment, but needs to be able to run local K8s without issue.
Any news on this? Having to recompile the WSL2 kernel to have a good working environment for k8s is annoying and consumes a good amount of time unless you have a powerful machine. It would be great to include this in the default build.
For future reference:
The following doc article about running Kind on WSL2 explains how to build your kernel to support session affinity: https://kind.sigs.k8s.io/docs/user/using-wsl2/#kubernetes-service-with-session-affinity