k3s: x509: certificate has expired or is not yet valid

I use this workaround to get the k3s-serving certificate renew.

kubectl --insecure-skip-tls-verify delete secret -n kube-system k3s-serving
rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json

systemctl restart k3s

Validated on version v1.24.4-rc1+k3s1

$ k3s -v
k3s version v1.24.4-rc1+k3s1 (c3f830e9)
go version go1.18.1

Environment Details

Infrastructure

• [ x ] Cloud

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"

Cluster Configuration:

1 server

Testing Steps

1. Install K3S curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.24.4-rc1+k3s1 INSTALL_K3S_EXEC="server" sh -
2. Make sure all pods are running

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-b96499967-6qvbc                   1/1     Running     0          3m2s
kube-system   helm-install-traefik-crd-527jx            0/1     Completed   0          3m2s
kube-system   helm-install-traefik-hbkvs                0/1     Completed   1          3m2s
kube-system   local-path-provisioner-7b7dc8d6f5-6mrrm   1/1     Running     0          3m2s
kube-system   metrics-server-668d979685-x8wzw           1/1     Running     0          3m2s
kube-system   svclb-traefik-74309c0a-5f7kt              2/2     Running     0          2m46s
kube-system   traefik-7cd4fcff68-x4vb5                  1/1     Running     0          2m46s

1. See current time

$ date
Mon Aug 22 19:51:17 UTC 2022

1. Stop k3s

systemctl stop k3s

1. Change the OS date to after the certificate expires

$ sudo timedatectl set-ntp no
$ sudo date -s 20230808
Tue Aug  8 00:00:00 UTC 2023
$ date
Tue Aug  8 00:00:05 UTC 2023

1. Start K3S

sudo systemctl start k3s

$ kubectl --insecure-skip-tls-verify  get secret -n kube-system k3s-serving -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | grep Not
            Not Before: Aug 22 19:47:50 2022 GMT
            Not After : Aug  7 00:00:50 2024 GMT

Validation Results: Message Unable to connect to the server: x509: certificate has expired or is not yet valid:...is not longer present

$ kubectl get nodes -A 
NAME               STATUS   ROLES                       AGE    VERSION
ip-172-31-47-144   Ready    control-plane,etcd,master   350d   v1.24.4-rc1+k3s1

Problem exists on 1.23.9+k3s1 as well.

The workaround works:

kubectl --insecure-skip-tls-verify delete secret -n kube-system k3s-serving
sudo rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json
sudo systemctl restart k3s

I see a potential fix (https://github.com/k3s-io/k3s/pull/5951) has been merged to 1.23. When do you think a release with this fix in place will be made please?

Also is there a way I can install the master 1.23.x version so I can test and report back it’s fixed on a VM?

@xzycn please see the steps at https://github.com/k3s-io/k3s/issues/5163#issuecomment-1189750770

@LarsBingBong no, what you’re reporting does not have anything to do with the dynamic certificate expiring. Please open a new issue.

Ah, I see. The locally cached secret is used initially, but when the apiserver comes up, the datastore secret is merged into it, and replaces it - which is what we want in general, to ensure that the datastore certificate is used in favor of the one from the local node. However, we shouldn’t do this if the one from the datastore is expired.