kubernetes: --log-file-max-size not getting honored by kubelet

What happened: I configured some nodes in a cluster to log to files and also specified --log-max-file-size and noticed that the log files grew significantly larger than what was specified.

What you expected to happen: Log files would rotate or trucate after the size exceeded the value specified with --log-max-file-size

How to reproduce it (as minimally and precisely as possible):

I was able to repro this on both Linux and Windows nodes by setting --logdir --logtostderr=flase --alsologtostderr --log-file-max-size as kubelet parameters.

In both cases I set --log-file-max-size=2 (the documentation says the flag is of type uint and in megabytes)

After running some work I observed the files mapped to kubelet.INFO / kubelet.exe.INFO growing to well paste 2Mb

on linux:

on windows:

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): 1.17.0
Cloud provider or hardware configuration: azure
OS (e.g: cat /etc/os-release): Ubuntu 16.04.6 LTS and Windows Server 2019
Kernel (e.g. uname -a):
Install tools:
Network plugin and version (if this is a network-related bug):
Others:

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (19 by maintainers)

Most upvoted comments

In brief, --log-file-max-size can only be useful for a specific file which is set by --log-file.

This logic in klog is :

func CalculateMaxSize() uint64 {
	if logging.logFile != "" {
		if logging.logFileMaxSizeMB == 0 {
			// If logFileMaxSizeMB is zero, we don't have limitations on the log size.
			return math.MaxUint64
		}
		// Flag logFileMaxSizeMB is in MB for user convenience.
		return logging.logFileMaxSizeMB * 1024 * 1024
	}
	// If "log_file" flag is not specified, the target file (sb.file) will be cleaned up when reaches a fixed size.
	return MaxSize
}

Which means, if there is not a log-file specified, then use the default max bytes: 1800M.

zhouya0 on Jan 13, 2020

/assign

yashikabadaya on Jun 25, 2021

/triage needs-information

Can someone please reproduce this?

I think there’s a few things we should do to follow up:

Validate if --log-file-max-size is set but --log-file is not set, and print a warning that the first flag will be ignored.
Improve documentation for these flags to indicate they both need to be used together.

I know there’s also some discussion of deprecating these flags.

/wg structured-logging

ehashman on Jun 24, 2021