kubernetes: Exceeding the ephemeral storage limit not causing eviction in critical pods

What happened?

As per my understanding, when containers exceed their memory or cpu limits, they are restarted. But when they exceed ephemeral storage limit, the pod itself has to be destroyed and recreated (by eviction). Also, critical pods would not be evicted. We have some pods marked as critical which at times consume ephemeral storage exceeding their limit (potentially filling up the disk). We have marked them as “critical” as we dont want them to become victim in case of any node pressure. At the same time, we would not want the pod to be evicted when they cross the configured limit. Is there any way we could achieve this?

What did you expect to happen?

Pods (even critical ones) have to be evicted when they consume more storage than the configured ephemeral storage limit.

How can we reproduce it (as minimally and precisely as possible)?

We can reproduce the issue by the following,

  1. Define ephemeral limit for any container in a pod.
  2. Mark the pod as critical by using setting priority class (priorityClassName: system-cluster-critical)
  3. Consume disk space from the container using dd if=/dev/urandom of=test or alternately configure very small ephemeral limit.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

v1.24.2

Cloud provider

Openstack

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

containerd github.com/containerd/containerd v1.6.6

Related plugins (CNI, CSI, …) and versions (if applicable)

https://github.com/kubernetes/kubernetes/issues/112893#tasklist-block-51c08819-e975-4a40-9758-ffcfe209ac8a

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 19 (9 by maintainers)

Most upvoted comments

/remove-kind bug /kind feature

@brucelee138 is there specific reason you want the pod to be marked critical. In general critical pods needs to be avoided