istio: CNI race condition alternative design
The CNI startup race condition (https://docs.google.com/document/d/1SQzrFxtcn3o_79OtJYbSHMPuqZNhR3EaEhbkpBVMXAw/edit#) is an issue that has plagued Istio for quite some time.
We currently have 2 solutions:
- init container on sidecar injected which tests rules are applied; if they aren’t we hang an a controller kills the pod. This is slow, gives high priv to controller, inefficient to kill pods, and confusing to users
- Taint controller; users need to configure their k8s to apply a taint by default to nodes and then we remove it. This works great but requires changes to the k8s cluster that we cannot do (no API)
The alternative I am thinking of is essentially the inverse of the taint controller. We inject a nodeSelector: cni.istio.io/ready=true onto all injected pods. Then, when the cni is ready we run the equivilent of kubectl label node cni.istio.io/ready=true. With this done, we can remove the istio-validation container entirely.
cc @stewartbutler @jwendell for opinions
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 2
- Comments: 28 (24 by maintainers)
https://github.com/kubernetes/kubernetes/pull/113274 is like scheduler idea kind of. But it’s alpha I think
This is an important point - this means we can’t hold app starting for anything related to waypoint.
https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/3521-pod-scheduling-readiness is an interesting find. We could use the scheduling gate to block the pods from being scheduled until we deemed it ready. Might be an overkill though since we might need a separate controller or something and it won’t be ready till k8s 1.27.
Hm, with my node selector thing Cluster autoscaler seems to work on GKE.