kind: HA clusters don't reboot properly

https://github.com/kubernetes-sigs/kind/issues/2045 (multi-node restart, not multi-control-plane-node) should be fixed at HEAD thanks to the patient work of @tnqn, have not dug into multi-control-plane node yet. Probably won’t be able to immediately, perhaps next week.

Hi @BenTheElder I know that using DNS names is the cleanest solution for issue #2045

However I am using this script as a workaround to use static IPs for the nodes communication

I have restarted my cluster several times and it has worked fine so far

Still needs root causing, but multiple user reports. We should fix this.

We document this sort of thing at https://kind.sigs.k8s.io/docs/user/known-issues/ which the quick start links to prominently, but it seems this issue hasn’t made it there yet. Earlier versions did not support host restart at all, it wasn’t in scope early in the project.

can’t we extend the kind config to take ip-node mapping as an optional parameter: to only those who know what they are doing. It can then completely replace every solution to perfectly fit in all needs.

This is not without its own drawbacks.

1. Not necessarily portable across node backends (e.g. out of our current options podman cannot do at least ipv6 this way).
2. Does not solve the need to set a reserved IP range in the kind network (so you will still need to do hacks outside of the kind tool…)
3. Adds infrequently used and untested codepath(s). (We are not going to add yet another CI job to exercise this, we have too many as-is and we have no need for this upstream https://kind.sigs.k8s.io/docs/contributing/project-scope/).

Multi-node clusters are a necessity for testing Kubernetes itself (where we expect clusters to be disposable over the course of developing some change to Kubernetes). For development of applications, we expect single node clusters to be most reasonable (and this is the case where it may make sense to persist them, though we’d still encourage regularly testing from a clean state).

The case of:

1. Requirement for multi-node
2. Requirement for persistence
3. Frequent reboots

Seems rather rare and I’m not sure it outweighs adding a broken partial solution that people will then depend on in the future even if we find some better design.

I’m not saying we definitely couldn’t do this, but I wouldn’t jump to doing it today.

That’s a neat script!

It’s unfortunately not super workable as an approach to a built-in solution though. Users creating clusters concurrently in CI (and potentially with a “remote” daemon due to containerized CI) are very important to us and this approach is not safe there.

I have restarted my cluster several times and it has worked fine so far

Users may have multiple clusters and that is hard to support, however, your script is great, I think that it also can solve the problems of snapshotting HA clusters.

@velcrine that’s actually a variation on the issues in https://github.com/kubernetes-sigs/kind/issues/2045

HA has a different additional problem in that the loadbalancer causes issues with the API being reachable after restart, in which case you wouldn’t even be able to query for those problems.

FWIW regarding HA Nobody is working on or using this feature much and it’s simplistic / not fully designed. This issue is unlikely to see work anytime soon. (priority/backlog)

The other issue (https://github.com/kubernetes-sigs/kind/issues/2045) is one I’m sure someone would work on except nobody has posited a good solution we can agree on yet or root caused the issues.