kubernetes: kubelet and kube-proxy fail to reload certificates when they are updated

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): tls certificate reload

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-25T14:48:12Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1+coreos.0", GitCommit:"9212f77ed8c169a0afa02e58dce87913c6387b3e", GitTreeState:"clean", BuildDate:"2017-04-04T00:32:53Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: Digital Ocean / custom setup
OS (e.g. from /etc/os-release): coreos VERSION=1353.7.0
Kernel (e.g. uname -a): Linux coreos01.kub.do.modio.se 4.9.24-coreos #1 SMP Wed Apr 26 21:44:23 UTC 2017 x86_64 Intel® Xeon® CPU E5-2650L v3 @ 1.80GHz GenuineIntel GNU/Linux
Install tools: Ansible / CoreOS getting started guide
Others:

What happened: We came up to our scheduled update of TLS client certificate, TLS certs were updated properly, but kubelet and kube-proxy keep the old Certs in memory. This causes them to fail when communicating with APIserver.

What you expected to happen: kubelet & kube proxy should reload the certificates from disk.

How to reproduce it (as minimally and precisely as possible): Generate a cert with a short lifetime, set up your cluster, wait a while, and then replace the cert with a longer lived one.

Anything else we need to know: We’re attempting to run with short lived client certificates. This has shown some issues with how kubernetes handles it, and will likely cause others to have hard to debug problems in the future.

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 9
Comments: 43 (27 by maintainers)

Most upvoted comments

/reopen

dims on Aug 1, 2019

Maybe we can big-hammer this? If the file changes, os.Exit() - it’s clumsy but it should work?

Help Wanted

thockin on Mar 2, 2023

Experiencing the same issue.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-12T00:44:36Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1+coreos.0", GitCommit:"59359d9fdce74738ac9a672d2f31e9a346c5cece", GitTreeState:"clean", BuildDate:"2017-10-12T21:53:13Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

alkar on Nov 3, 2017