kubernetes: kubelet doesn't clean up the termination-log file on the host

What happened?

Each time a container needs to be created, the kubelet will create a termination-log file on the host and mounts the file to the path specified by the terminationMessagePath field in the container for outputting termination messages. The paths of these files on the host are as follows: /var/lib/kubelet/pods/{POD_ID}/containers/{CONTAINER_NAME}/{RANDOM_NUM}

It seems that kubelet does not clean up these files unless the pod is deleted. If some containers are restarted for many times, a large number of files may be generated and the host inodes are used up.

What did you expect to happen?

kubelet should proactively clear these files to avoid inode exhaustion.

How can we reproduce it (as minimally and precisely as possible)?

Re-create a running container and you can see the newly created file in the path: /var/lib/kubelet/pods/{POD_ID}/containers/{CONTAINER_NAME}/{RANDOM_NUM}

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
v1.22.1

Cloud provider

Not relevant

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

About this issue

Original URL
State: open
Created 3 years ago
Comments: 16 (11 by maintainers)

Most upvoted comments

@rphillips @luozhiwenn

The kubelet creates a log file with the ${restartCount}.log; not a random number. When the kubelet rotates the log, then old log files are cleaned up. After the pod is removed from the disk (via a GC or API delete), then all the pod logs are removed.

I already reported about this in #114117 and I report it again in this issue.

I can see the two kind log.

Termination-log: /var/lib/kubelet/pods/{POD_ID}/containers/{CONTAINER_NAME}/{RANDOM_NUM}
Container-log: /var/log/pods/{NAMESPACE}_{POD_NAME}_{POD_ID}/{CONTAINER_NAME}/{restartCount}.log

The container-log is not a random number, but the termination-log is a random number. I think this issue is about termination-log.

When a container-log is roteted, an old container-log is removed in

https://github.com/kubernetes/kubernetes/blob/4c4d4ad0a4aea4d015561ae3e7d48e8aaf609277/pkg/kubelet/kuberuntime/kuberuntime_container.go#L971-L973

However, even if a termination-log is rotated, an old termination-log is not removed.

If my report is correct, I think this Issue should be continued. What do you think?

s-kawamura-w664 on Jan 16, 2023