kube-router: kube router generates incorrect ipsets

What happened? after upgrading from kube router 1.1.1 to 1.2.2 the network communication on some nodes of our cluster suddenly stops after some time. we found out that the ipset for the local ips got messed up, all entries are gone. we checked the ipsets on the working nodes and there the ipset kube-router-local-ips has three or four members (depending on the node type).

What did you expect to happen? kube router should not discard the members in the ipset kube-router-local-ips

How can we reproduce the behavior you experienced? We cannot say how this problem can be reproduced, since from our point of view it appears out of nowhere.

** Screenshots / Architecture Diagrams / Network Topologies **

Screenshot 2021-05-12 131044

** System Information (please complete the following information)😗*

  • Kube-Router Version 1.2.2
  • Kube-Router Parameters: --run-router=true --run-firewall=true --run-service-proxy=true --kubeconfig=/var/lib/kube-router/kubeconfig --metrics-port=9912
  • Kubernetes Version (kubectl version) : 1.20.5
  • Cloud Type: on premise
  • Kubernetes Deployment Type: custom setup
  • Kube-Router Deployment Type: DaemonSet
  • Cluster Size: 4

** Logs, other output, metrics ** No logs were recorded at the time the error occurred

Additional context

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (2 by maintainers)

Most upvoted comments

After 5+ days of testing in multiple clusters and these ipset symptoms not resurfacing I have merged #1088 and cut a kube-router bugfix release that contains this and several other fixes.

For anyone experiencing this issue, please try kube-router-1.2.3: