minikube: Renew cert on start if current cert has has expired

Steps to reproduce the issue:

Oracle vm virtualbox version 6.1
kubectl version Client Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.1”, GitCommit:“c4d752765b3bbac2237bf87cf0b1c2e307844666”, GitTreeState:“clean”, BuildDate:“2020-12-19T07:38:38Z”, GoVersion:“go1.15.5”, Compiler:“gc”, Platform:“darwin/amd64”}

minikube version minikube version: v1.16.0 commit: 9f1e482427589ff8451c4723b6ba53bb9742fbb1

$ docker --version Docker version 20.10.2, build 2291f61

minikube start 😄 minikube v1.16.0 on Darwin 10.15.7 🆕 Kubernetes 1.20.0 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.20.0 ✨ Using the virtualbox driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🏃 Updating the running virtualbox “minikube” VM … 🐳 Preparing Kubernetes v1.17.0 on Docker 19.03.5 …| E0110 11:59:25.743916 9686 kubeadm.go:647] sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml failed - will try once more: /bin/bash -c “sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml”: Process exited with status 1 stdout: [certs] Using certificateDir folder “/var/lib/minikube/certs” [certs] Using existing ca certificate authority [certs] Using existing apiserver certificate and key on disk

stderr: W0110 16:59:25.557503 17370 validation.go:28] Cannot validate kube-proxy config - no validator is available W0110 16:59:25.557541 17370 validation.go:28] Cannot validate kubelet config - no validator is available error execution phase certs/apiserver-kubelet-client: failed to write or validate certificate “apiserver-kubelet-client”: failure loading apiserver-kubelet-client certificate: failed to load certificate: the certificate has expired To see the stack trace of this error execute with --v=5 or higher / 🤦 Unable to restart cluster, will reset it: run: /bin/bash -c “sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml”: Process exited with status 1 stdout: [certs] Using certificateDir folder “/var/lib/minikube/certs” [certs] Using existing ca certificate authority [certs] Using existing apiserver certificate and key on disk

stderr: W0110 16:59:25.744187 17376 validation.go:28] Cannot validate kube-proxy config - no validator is available W0110 16:59:25.744235 17376 validation.go:28] Cannot validate kubelet config - no validator is available error execution phase certs/apiserver-kubelet-client: failed to write or validate certificate “apiserver-kubelet-client”: failure loading apiserver-kubelet-client certificate: failed to load certificate: the certificate has expired To see the stack trace of this error execute with --v=5 or higher

▪ Generating certificates and keys .../ 💢  initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap": Process exited with status 1

stdout: [init] Using Kubernetes version: v1.17.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’ [kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env” [kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml” [kubelet-start] Starting the kubelet [certs] Using certificateDir folder “/var/lib/minikube/certs” [certs] Using existing ca certificate authority [certs] Using existing apiserver certificate and key on disk

stderr:

▪ Generating certificates and keys ...-

💣 Error starting cluster: wait: /bin/bash -c “sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable–etc-kubernetes-manifests,DirAvailable–var-lib-minikube,DirAvailable–var-lib-minikube-etcd,FileAvailable–etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable–etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable–etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable–etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap”: Process exited with status 1 stdout: [init] Using Kubernetes version: v1.17.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’ [kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env” [kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml” [kubelet-start] Starting the kubelet [certs] Using certificateDir folder “/var/lib/minikube/certs” [certs] Using existing ca certificate authority [certs] Using existing apiserver certificate and key on disk

stderr:

😿 minikube is exiting due to an error. If the above message is not useful, open an issue: 👉 https://github.com/kubernetes/minikube/issues/new/choose

❌ Exiting due to GUEST_START: wait: /bin/bash -c “sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable–etc-kubernetes-manifests,DirAvailable–var-lib-minikube,DirAvailable–var-lib-minikube-etcd,FileAvailable–etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable–etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable–etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable–etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap”: Process exited with status 1 stdout: [init] Using Kubernetes version: v1.17.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’ [kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env” [kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml” [kubelet-start] Starting the kubelet [certs] Using certificateDir folder “/var/lib/minikube/certs” [certs] Using existing ca certificate authority [certs] Using existing apiserver certificate and key on disk

stderr:

😿 If the above advice does not help, please let us know: 👉 https://github.com/kubernetes/minikube/issues/new/choose

Optional: Full output of minikube logs command:

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 9
Comments: 28 (7 by maintainers)

Most upvoted comments

Also ran into this issue earlier today. However, it persists with the same error even if I downgrade minikube with this version for instance:

minikube v1.14.2 on Darwin 10.15.7

Same stdout and stderr output referencing to the certificate issue. EDIT: solved it by deleting the cluster with minikube delete and start a new one but that does not reveal the underlying cause of this issue.

+17

dabcoder on Jan 11, 2021

There’a a workaround for those who want to keep the cluster data. Once the minikube vm refuse to start, ssh into it and go to /var/lib/minikube/certs. I deleted the certificates that were reported as expired (don’t have the list as I processed them incrementally, maybe deleting the certs folder may work) as they will be re created

Deleting the whole certs folder did not help. I restored it and then deleted the problematic certs iteratively, and in the end got it working, but all deployments were lost, so might have as well deleted minikube. In case it might be useful to somebody, here is what I did: minikube start failed similar to the OP Then I did

minikube ssh
su - root
cp -r /var/lib/minikube/certs /var/lib/minikube/certs.back
rm /var/lib/minikube/certs/etcd/server*
rm /var/lib/minikube/certs/etcd/peer*
rm /var/lib/minikube/certs/etcd/healthcheck*
rm /var/lib/minikube/certs/apiserver-etcd*
exit
exit
minikube stop
minikube start

This results in:

* minikube v1.9.2 on Microsoft Windows 10 Enterprise 10.0.17763 Build 17763
* Using the hyperv driver based on existing profile
* Starting control plane node m01 in cluster minikube
* Restarting existing hyperv VM for "minikube" ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
! Unable to restart cluster, will reset it: apiserver health: controlPlane never updated to v1.18.0
* Enabling addons: default-storageclass, storage-provisioner
! Enabling 'default-storageclass' returned an error: running callbacks: [Error making standard the default storage class: Error listing StorageClasses: Unauthorized]
* Done! kubectl is now configured to use "minikube"

Note that kubectl get nodes would return error: You must be logged in to the server (Unauthorized) In my case, kubectl was configured with:

- name: minikube
  user:
    client-certificate: C:\Users\User\.minikube\profiles\minikube\client.crt
    client-key: C:\Users\User\.minikube\profiles\minikube\client.key

These certs correspond to /var/lib/minikube/certs/apiserver.[crt|key] inside the minikube VM. So I got them with:

minikube ssh
su - root
cat /var/lib/minikube/certs/apiserver.crt
cat /var/lib/minikube/certs/apiserver.key
exit
exit
minikube stop
minikube start

And replaced the contents of client.crt and client.key in C:\Users\User\.minikube\profiles\minikube\ accordingly. Then kubectl get nodes started working.

But, as I noted above, the cluster got reset at some point, so I had to redeploy everything, which is a big problem for using minikube for anything other than a quick-dev/test and then throw it away. We need to be able to set it up, configure and leave it working for years without having to worry about it resetting once certificates expire.

edemen on Jul 28, 2021

in my case deleting rm ~/.minikube/client.{crt,key} files and then minikube delete worked fine, i was okay to delete minikube vm

infa-ddeore on Jul 29, 2021

I ran into this issue today, deleting /var/lib/minikube/certs did not help. Is there any other workaround, please?

Osprey2021 on Jul 9, 2021

Slightly more surgical version of “delete all files and folders in /var/lib/minikube/certs” that worked for me in the case where I could not even start the cluster:

minikube ssh
cd /var/lib/minikube/certs
sudo find . -type f ! -mtime 2 -name '*.crt' -delete
sudo find . -type f ! -mtime 2 -name '*.key' -delete

This deletes all the certs and keys that were last modified 2 or more days ago.

mbaynton on Dec 21, 2023

I delete files and folders in the “/var/lib/minikube/certs/*” and minikube stop then start fix my issue

LeBakii on Dec 8, 2022

Can you

minikube ssh

cd var/lib/minikube/certs

ls -l

All files listed should have yesterday or today’s date

Some files in that folder have new date, and some have older date. I moved certs folder to certs_BC so that certs folder was recreated, so am not sure how its possible that some older files reappear here.

Osprey2021 on Jul 13, 2021