kubernetes: Pod with dynamic volume stays in 'unknown' state if node powers off. Delete not possible
Hi,
we are using rook (ceph) as storage solution in our kubernetes cluster.
If we deploy the mysql demo from rook everything works as expected. But if we kill the node which hosts the mysql pod, the mysql pod gets this status, the newly spawned pod is stuck in ContainerCreating forever:
default wordpress-mysql-5f4b68789d-fsxhs 1/1 Unknown 0 10m 10.42.5.9 chmu007w001
default wordpress-mysql-5f4b68789d-sqflx 0/1 ContainerCreating 0 54s <none> chmu004m002
Deletion of the pod with this method doesn’t work:
kubectl delete pod wordpress-mysql-5f4b68789d-fsxhs --force --grace-period=0 --wait=false
The pod with the ‘unknown’ pod will be still there, the newly spawned pod is stuck in ‘ContainerCreating’.
Shouldn’t take Kubernetes take care about that we can delete also PODs which are on lost nodes?
Thanks and best regards,
Josef
@kubernetes/sig-scheduling-bugs @kubernetes/sig-storage-bugs @kubernetes/sig-node-bugs
- Kubernetes version (use
kubectl version): 1.10.3 - Cloud provider or hardware configuration: VM
- OS (e.g. from /etc/os-release): CentOs 7.5
- Kernel (e.g.
uname -a): 3.10.0-862.2.3.el7.x86_64 - Install tools: Rancher RKE
- Others:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 19 (3 by maintainers)
This stackoverflow article sent me in the right direction and solved my problem:
https://stackoverflow.com/questions/35453792/pods-stuck-at-terminating-status
I had to type:
An editor opens with the manifest for the pod. Remove these two lines:
Saved the file and leave the editor. The pod with the ‘unknown’ state will be killed immediately and the newly spawned pod will start as usual.
Shouldn’t Kubernetes take care about that?
I mean: ‘force’ should tell kubernetes that I really really really want to kill a pod. Isn’t it 😃 ?
Josef