autoscaler: Annotation cluster-autoscaler.kubernetes.io/safe-to-evict=false not working
I recently added the following annotation in some of my critical pods to avoid cluster-autoscaler to remove the nodes where these pods are running in. cluster-autoscaler.kubernetes.io/safe-to-evict=false
According to what I read in the documentation: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node This should prevent cluster autoscaler from removing the node, however I’ve just seen how one of the nodes running a pod with this annotations has been deleted by cluster autoscaler.
My Kubernetes version is 1.11 and the cluster-autoscaler version is 1.3.7 According to this PR (https://github.com/kubernetes/autoscaler/pull/1054), this annotation should work as expected.
Could someone help me to identify why cluster-autoscaler does not take this annotation in consideration?
Thanks!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 25 (10 by maintainers)
We could found out the problem in our case. After saving all logs from cluster-autoscaler in our logging system, we could do some more in depth investigation about this problem and it turned out to be an issue produced by AZRebalance in EC2 instances. Sometimes after cluster-autoscaler started to terminate instances, we were running more machines in certain AZ than the others, therefore AWS started to terminate instances from this AZ to rebalance the number of machines running in each AZ and since this is a process in EC2, any machine could be terminated regardless what’s running in. And that was exactly the reason why instances running coreDNS pods where terminating.
Sorry for the inconvenience of this issue and thanks a lot for your support.
@bskiba Yes, that was exactly the problem.
Thanks for the followup! The behaviour seems to be documented here: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#common-notes-and-gotchas is this in line with what you experienced?