kubeadm: Kubelet can't running after renew certificates

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:27:49Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Cloud provider or hardware configuration: bare-metal
OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):

Linux k8s-236 3.10.0-957.12.2.el7.x86_64 #1 SMP Mon May 20 08:41:20 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Others:

What happened?

I using the kubeadm command to renew the cluster certificates, the kubeadm alpha certs renew all works well, and i found the certificates have been changed. But when restart kubelet, kubelet can’t running, it exited. I don’t know why? How to do renew control plane certficates and kubelet certficates, which docs can i reference.

What you expected to happen?

Renew all cluster certificates successfully and kubelet running.

How to reproduce it (as minimally and precisely as possible)?

create cluster via kubeadm.
enable kubelet dynamic config.
update the host datetime, make the certificates expiration.
run kubeadm alpha certs renew all command
systemctl restart kubelet

Anything else we need to know?

kubelet log:

server.go:821] Client rotation is on, will bootstrap in background
bootstrap.go:265] part of the existing bootstrap client certificate is expired: 2021-03-04 14:19:56 +0000 UTC
systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
server.go:273] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 24 (12 by maintainers)

Most upvoted comments

@neolit123 thanks for the reply.

The fix for me was to copy the contents of /etc/kubernetes/admin.conf specifically the keys client-certificate-data and client-key-data and paste those new strings into the file /etc/kubernetes/kubelet.conf under the same keys. Then just a sudo service kubelet restart.

Why is this not more obvious when invoking kubeadm alpha certs renew all? If it’s not going to update kubelet.conf there should be a warning message or something. This costs me a few hours last night of time and downtime.

+18

nodesocket on Feb 4, 2022

instead of copying the cert/key inside the kubeconfig, see what is suggested here: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration

Warning: On nodes created with kubeadm init, prior to kubeadm version 1.17…

neolit123 on Mar 31, 2020

The fix for me was to copy the contents of /etc/kubernetes/admin.conf specifically the keys client-certificate-data and client-key-data and paste those new strings into the file /etc/kubernetes/kubelet.conf under the same keys. Then just a sudo service kubelet restart.

this is not ideal, as it grants the kubelet client super admin credentials…you are also hardcoding non-rotatable credentials. the link i’ve mentioned has steps to generate better scoped credentials: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert

Why is this not more obvious when invoking kubeadm alpha certs renew all? If it’s not going to update kubelet.conf there should be a warning message or something. This costs me a few hours last night of time and downtime.

this is documented: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

kubelet.conf is not included in the list above because kubeadm configures kubelet for automatic certificate renewal with rotatable certificates under /var/lib/kubelet/pki. To repair an expired kubelet client certificate see Kubelet client certificate rotation fails.

but back to the original problem, for the credentials to expire something must have gotten wrong on the node…or you had hardcoded kubelet.conf credentials instead of pointing to the rotatable symlink (explained in the docs). the kubelet rotates the symlinked files every ~8 months.

EDIT: also check this warning in the docs:

On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

neolit123 on Feb 5, 2022

I manually copied the renewed cert and key from /var/lib/kubelet/pki/kubelet-client-current.pem to /etc/kubernetes/kubelet.conf, and it started. Phew!

dimm0 on Mar 31, 2020