kubernetes: Exceeding the ephemeral storage limit not causing eviction in critical pods
What happened?
As per my understanding, when containers exceed their memory or cpu limits, they are restarted. But when they exceed ephemeral storage limit, the pod itself has to be destroyed and recreated (by eviction). Also, critical pods would not be evicted. We have some pods marked as critical which at times consume ephemeral storage exceeding their limit (potentially filling up the disk). We have marked them as “critical” as we dont want them to become victim in case of any node pressure. At the same time, we would not want the pod to be evicted when they cross the configured limit. Is there any way we could achieve this?
What did you expect to happen?
Pods (even critical ones) have to be evicted when they consume more storage than the configured ephemeral storage limit.
How can we reproduce it (as minimally and precisely as possible)?
We can reproduce the issue by the following,
- Define ephemeral limit for any container in a pod.
- Mark the pod as critical by using setting priority class (priorityClassName: system-cluster-critical)
- Consume disk space from the container using dd if=/dev/urandom of=test or alternately configure very small ephemeral limit.
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
# paste output here
v1.24.2
Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, …) and versions (if applicable)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 19 (9 by maintainers)
Exceeding the limit is a condition for triggering eviction. After the eviction starts, it will be judged whether it is a critical pod. https://github.com/kubernetes/kubernetes/blob/594d34872c73f6abe5eba740f9e4557c6f970887/pkg/kubelet/eviction/eviction_manager.go#L505
https://github.com/kubernetes/kubernetes/blob/594d34872c73f6abe5eba740f9e4557c6f970887/pkg/kubelet/eviction/eviction_manager.go#L520
/remove-kind bug /kind feature
@brucelee138 is there specific reason you want the pod to be marked critical. In general critical pods needs to be avoided