kubernetes: Rollback cordon if kubectl drain fails

kubectl can fail for a couple of reasons and we apparently leave the node cordoned

$ k drain e2e-test-beeps-minion-group-iinp
node "e2e-test-beeps-minion-group-iinp" cordoned
error: pods with local storage (use --delete-local-data to override): busybox-uid, zoo-0; pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet (use --force to override): busybox-uid, default-http, host-test-container-pod, netserver-1, netserver-1, test-container-pod, netserver-0, test-container-pod, host-test-container-pod, netserver-0, fluentd-cloud-logging-e2e-test-beeps-minion-group-iinp, kube-proxy-e2e-test-beeps-minion-group-iinp; DaemonSet-managed pods (use --ignore-daemonsets to ignore): node-problem-detector-v0.1-yifp5

$ k drain e2e-test-beeps-minion-group-iinp --delete-local-data
node "e2e-test-beeps-minion-group-iinp" already cordoned
...

So I guess we’re leaking a cordon?

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 2
Comments: 19 (17 by maintainers)

Commits related to this issue

Don't cordon if kubectl drain won't succeed Instead of draining the node then realising that we have a pod with an EmptyDir or similar, run a preflight check to see if any such pods exist first. Sin... — committed to lukegb/kubernetes by lukegb 7 years ago

Most upvoted comments

IMO it would be better to just not cordon when we know 100% we’re not going to finish the drain (–delete-local-data case can be detected beforehand)

bprashanth on Dec 12, 2016