calico: Incompatible ipset protocol version (7) will cause problems
Expected Behavior
Calico can handle different ipset protocol versions. At least 6-7.
Current Behavior
If K8s upgrades to ipset >= v7.0 outgoing connects from pods fail.
Possible Solution
Upgrade the calico image to include ipset v7.x since it’s backward compatible.
Steps to Reproduce (for bugs)
- Make sure K8s is upgraded to use
ipsetv7.x (this is non-trivial if installed with kubeadm) - Use proxy-mode=ipvs (uses ipset)
- Try an outgoing connect from a pod
The kube-proxy (v1.22.2) image includes ipset v6.38. It must be changed to some v7.x version. How that is done I can’t say. I start kube-proxy as a program (no container) so for me it’s simply to change the ipset on the node.
Context
While this isn’t a problem at the moment, it will be the day K8s upgrades to iset v7.x
Your Environment
- Calico version: calico/cni:v3.19.1
- Orchestrator version (e.g. kubernetes, mesos, rkt): K8s
- Operating System and version: Own. But Ubuntu 20.04.3 LTS uses ipset v7.5 at the moment
- Link to your project (optional):
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 8
- Comments: 38 (16 by maintainers)
Commits related to this issue
- kola/test/kubeadm: temporary exclude calico on alpha kernel 5.15 and calico are currently clashing. See also: https://github.com/projectcalico/calico/issues/5011 Signed-off-by: Mathieu Tortuyaux <mt... — committed to flatcar/mantle by tormath1 2 years ago
- Update Calico from v3.21.2 to v3.22.1 * Calico aims to fix https://github.com/projectcalico/calico/issues/5011 — committed to poseidon/terraform-render-bootstrap by dghubble 2 years ago
- Update Calico from v3.21.2 to v3.22.1 * https://github.com/projectcalico/calico/releases/tag/v3.22.1 * Should fix https://github.com/projectcalico/calico/issues/5011 — committed to poseidon/typhoon by dghubble 2 years ago
- Update Calico from v3.21.2 to v3.22.1 * https://github.com/projectcalico/calico/releases/tag/v3.22.1 * Fix https://github.com/projectcalico/calico/issues/5011 — committed to poseidon/typhoon by dghubble 2 years ago
- Update Calico from v3.21.2 to v3.22.1 * https://github.com/projectcalico/calico/releases/tag/v3.22.1 * Fix https://github.com/projectcalico/calico/issues/5011 — committed to aristanetworks/monsoon by dghubble 2 years ago
- calico-node: backport ipset to v3.12.1 FROM following references: - Incompatible ipset protocol version (7) will cause problems - https://github.com/projectcalico/calico/issues/5011 - Update Node ... — committed to yunionio/container-images by zexi 6 months ago
- calico-node: backport ipset to v3.12.1 FROM following references: - [Incompatible ipset protocol version (7) will cause problems](https://github.com/projectcalico/calico/issues/5011) - [Update Node ... — committed to yunionio/container-images by zexi 6 months ago
Kubernetes v1.23.0
kube-proxydoes use v.7.x now. Calico v3.21.x Pods show the OP’s error and will never be Ready. So the time has come.That’s probably because you’re using proxy-mode=iptables
FYI: The ipset changes is not part of v3.22.0 since it takes more time to get everything fully tested. It should be in v3.22.1.
Interesting, this kernel version doesn’t include bucketsize in all hash types for the ipset kernel module that’s why you seem to have no issues.
EDIT : Kernel >= 5.11 includes bucketsize in all hash types.
ipvs proxy mode SHOULD be disabled to make Calico work on kubernetes >= 1.23 AND host kernel >= 5.11
Can you check calico-node logs and ipset version on host VS on calico-node pod ?
I got errors when using ipvs mode for dualstack use, iptables is fine on my 1.23 cluster