metrics-server: x509: certificate signed by unknown authority (--kubelet-insecure-tls didn't help)

Hello, on baremetall Kubernetes version 1.16.2, I’m trying install metrics-server v0.3.6 and get these errors in metrics-server logs

E0320 08:04:20.914963 1 authentication.go:65] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "front-proxy-ca"), x509: certificate signed by unknown authority] –kubelet-insecure-tls parameter didn’t help

/king bug

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (6 by maintainers)

Most upvoted comments

I am experiencing the same issue when I use the self signed certificate. I have tried specifying the front-proxy-ca.crt from a config map, but then the service is unresponsive.

# on master where the crt is located
kubectl -n kube-system create configmap front-proxy-ca --from-file=/etc/kubernetes/pki/front-proxy-ca.crt

# Edit the deployment and mount the cert
spec:
	containers:
		 volumeMounts:
	        - name: front-proxy-ca
	          mountPath: /front-proxy-ca.crt
	          subPath: front-proxy-ca.crt
	          readOnly: true
	volumes:
	      - name: front-proxy-ca
	        configMap:
	          name: front-proxy-ca

# Also pass the file path to the metrics server command
spec:
	command:
          - /metrics-server
          - --cert-dir=/tmp
          - --logtostderr
          - --secure-port=8443
          - --requestheader-client-ca-file=/front-proxy-ca.crt

After specifying the crt, everything looks ok from the pod logs:

$ k logs pod/metrics-server-5cc8997d59-2sn4t
I0420 02:37:40.257883       1 secure_serving.go:116] Serving securely on [::]:8443

but describing it shows the service is unhealthy

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  7m10s                  default-scheduler  Successfully assigned kube-system/metrics-server-5cc8997d59-2sn4t to delta
  Normal   Killing    2m52s (x2 over 4m52s)  kubelet, delta     Container metrics-server failed liveness probe, will be restarted
  Normal   Pulled     2m51s (x3 over 7m8s)   kubelet, delta     Container image "k8s.gcr.io/metrics-server-arm:v0.3.6" already present on machine
  Normal   Created    2m51s (x3 over 7m7s)   kubelet, delta     Created container metrics-server
  Normal   Started    2m51s (x3 over 7m6s)   kubelet, delta     Started container metrics-server
  Warning  Unhealthy  62s (x8 over 5m12s)    kubelet, delta     Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  55s (x7 over 5m15s)    kubelet, delta     Readiness probe failed: HTTP probe failed with statuscode: 500

k top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

Other details: helm, arm, kubeadm, weave, single master

I don’t see any other errors save the unhealthy probe, so I am unsure how I can diagnose this further.

kluzny on Apr 20, 2020