kubernetes: kube-proxy fails to delete nat entries with IPv6
What happened:
kube-proxy fails to clear IPv6 conntrack entries
What you expected to happen:
conntrack entries should be cleared
How to reproduce it (as minimally and precisely as possible):
Deploying a cluster with kind and IPv6 and restarting the coredns pods
Error in kube-proxy
conntrack command returned: "conntrack v1.4.4 (conntrack-tools): mismatched address family\nTry `conntrack -h' or 'conntrack --help' for more information.\n", error message: exit status 2
E0821 15:22:29.549792 1 proxier.go:635] Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {fd00:10:96::a, fd00:10:244::3}, error: conntrack command returned: "conntrack v1.4.4 (conntrack-tools): mismatched address family\nTry `conntrack -h' or 'conntrack --help' for more information.\n", error message: exit status 2
Running conntrack from the host
root@kind-control-plane:/# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3 -p udp
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been deleted.
root@kind-control-plane:/# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3 -p tcp
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been deleted.
root@kind-control-plane:/# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been deleted.
Running conntrack from the kube-proxy container
root@kind-control-plane:/# crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
da8c90d79a13c 7987f0908caf3 2 hours ago Running coredns 0 b76ad55de5756
fd2be218f5227 7987f0908caf3 2 hours ago Running coredns 0 024d498118bf2
c270083cac0a1 ef97cccdfdb50 2 hours ago Running kindnet-cni 0 afa0a0e5310ef
09539af960455 d6f7fd86934c2 2 hours ago Running kube-proxy 0 c8fd75dced320
9a468e0cac7b4 2c4adeb21b4ff 2 hours ago Running etcd 0 1c44a4fe1d33a
978f3d9319e49 b599d33ed5a7b 2 hours ago Running kube-controller-manager 0 3ec62550d5915
e22fe1f8bbf4c 2b7262d4b0fbb 2 hours ago Running kube-scheduler 0 09f0ac50a30e5
f6944e9451fd5 bb8dba7182b8a 2 hours ago Running kube-apiserver 0 356f94ddca189
root@kind-control-plane:/# crictl exec -it 09539af960455 sh
# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3 -p udp
conntrack v1.4.4 (conntrack-tools): mismatched address family
Try `conntrack -h' or 'conntrack --help' for more information.
Anything else we need to know?:
Seems to be a bug in the conntrack tools 1.4.4 per snippet pasted previously
Environment:
- Kubernetes version (use
kubectl version
): - Cloud provider or hardware configuration:
- OS (e.g:
cat /etc/os-release
): - Kernel (e.g.
uname -a
): - Install tools:
- Network plugin and version (if this is a network-related bug):
- Others:
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (14 by maintainers)
OK, I’ve reached out the package owner in Debian, but the solution is to update the container images to Debian 10.
I’ll take this to the next sig-network meeting to see how can we deal with this