kubernetes: kube-proxy fails to delete nat entries with IPv6

What happened:

kube-proxy fails to clear IPv6 conntrack entries

What you expected to happen:

conntrack entries should be cleared

How to reproduce it (as minimally and precisely as possible):

Deploying a cluster with kind and IPv6 and restarting the coredns pods

Error in kube-proxy

conntrack command returned: "conntrack v1.4.4 (conntrack-tools): mismatched address family\nTry `conntrack -h' or 'conntrack --help' for more information.\n", error message: exit status 2
E0821 15:22:29.549792       1 proxier.go:635] Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {fd00:10:96::a, fd00:10:244::3}, error: conntrack command returned: "conntrack v1.4.4 (conntrack-tools): mismatched address family\nTry `conntrack -h' or 'conntrack --help' for more information.\n", error message: exit status 2

Running conntrack from the host

root@kind-control-plane:/# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3 -p udp
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been deleted.
root@kind-control-plane:/# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3 -p tcp
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been deleted.
root@kind-control-plane:/# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been deleted.

Running conntrack from the kube-proxy container

root@kind-control-plane:/# crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
da8c90d79a13c       7987f0908caf3       2 hours ago         Running             coredns                   0                   b76ad55de5756
fd2be218f5227       7987f0908caf3       2 hours ago         Running             coredns                   0                   024d498118bf2
c270083cac0a1       ef97cccdfdb50       2 hours ago         Running             kindnet-cni               0                   afa0a0e5310ef
09539af960455       d6f7fd86934c2       2 hours ago         Running             kube-proxy                0                   c8fd75dced320
9a468e0cac7b4       2c4adeb21b4ff       2 hours ago         Running             etcd                      0                   1c44a4fe1d33a
978f3d9319e49       b599d33ed5a7b       2 hours ago         Running             kube-controller-manager   0                   3ec62550d5915
e22fe1f8bbf4c       2b7262d4b0fbb       2 hours ago         Running             kube-scheduler            0                   09f0ac50a30e5
f6944e9451fd5       bb8dba7182b8a       2 hours ago         Running             kube-apiserver            0                   356f94ddca189
root@kind-control-plane:/# crictl exec -it 09539af960455 sh
# conntrack -f ipv6 -D --orig-dst fd00:10:96::a --dst-nat fd00:10:244::3 -p udp
conntrack v1.4.4 (conntrack-tools): mismatched address family
Try `conntrack -h' or 'conntrack --help' for more information.

Anything else we need to know?:

Seems to be a bug in the conntrack tools 1.4.4 per snippet pasted previously

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (14 by maintainers)

Most upvoted comments

OK, I’ve reached out the package owner in Debian, but the solution is to update the container images to Debian 10.

I’ll take this to the next sig-network meeting to see how can we deal with this