kubernetes: Use of glog for logging is problematic

FEATURE REQUEST:

/kind feature

What happened:

Kubernetes uses golang/glog extensively for logging. While this is reasonable enough for the Kubernetes binaries, sizeable parts of the codebase are re-used as client libraries (eg. k8s.io/client-go). This presents several problems for users of the libraries:

glog registers a bunch of flags in an init function, and offers no programmatic way to configure its behaviour. It can be surprising to users of the k8s libraries that they have to call flag.Parse(), and this is easy to get wrong (eg. coredns/coredns#1597).
glog’s default behaviour is to log to a file on disk. A user of a library wouldn’t typically expect it to write files without explicitly configuring it to do so.
Worse, if glog cannot create a file, it calls os.Exit. This can be very harmful, and since it’s common to run containerised binaries with a read-only root filesystem, can be easy to trigger.
glog doesn’t perform any management of the files it writes, so without something like logrotate running (especially problematic in a container) the log files just accumulate.

I think we need to find a way to fix this: the use of glog makes working with the k8s client libraries very difficult. A recent Twitter discussion on this topic shows this is a widely-held concern.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 42
Comments: 23 (14 by maintainers)

Links to this issue

Most upvoted comments

2 more problems with glog I’d like to add:

Logs are untestable (by unit tests). There is no good way to get at the log file, or intercept logs.
No way to direct logs to a specific file. As a result, a lot of our system addons log to stderr, and then use a shell to redirect the logs to a file. This is problematic for a whole bunch of reasons.

Forking glog

+1. I think a sensible path forward would be:

Fork glog, address the immediate problems without changing the interface
(simultaneously) collect requirements & review existing log libraries to identify the library we want to migrate to
Reimplement our glog fork as an adapter to the new logging library
Gradually migrate log usage to the new library interface (or an alternative interface we design).

tallclair on Sep 24, 2018

Pinging some people who we’ve spoken to about this - @bboreham @munnerz - @thockin had some ideas in the hallway at KubeCon CPH about how to approach this as a piecemeal solution gradually replacing glog with an interface implemented by glog for a non-harm change.

alexellis on Jun 16, 2018

FYI, main k/k repo is now using k8s.io/klog

dims on Nov 10, 2018

I agree glog should be fixed, forked or replaced.
Note the glog README says “Feature requests will be ignored.”, which points towards the latter two.

Interested in your view @thockin

bboreham on Mar 14, 2018