metrics-server: metrics-server with k8s 1.14.9 eks.8: Unable to authenticate the request due to an error: x509: certificate signed by unknown authority

What happened: I get errors in metrics-server. See logs below. What you expected to happen: I expected metrics-server to work. Anything else we need to know?: This should be a very simple vanilla eks cluster setup with eksctl. I installed metrics-server using the simple instructions at https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html

I’ve done lots of searches and read through the following three tickets tickets which are somewhat related but not entirely related: https://github.com/kubernetes-sigs/metrics-server/issues/292 https://github.com/kubernetes-sigs/metrics-server/issues/146 https://github.com/kubernetes-sigs/metrics-server/issues/319

I’ve tried manually adding the --kubelet-insecure-tls argument to the deployment manifest and that didn’t seem to help at all.

Environment:

Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): EKS. Current version as of this writing. eks.8 1.14.9.
Container Network Setup (flannel, calico, etc.):
Kubernetes version (use kubectl version): v1.14.9-eks-502bfb
Metrics Server manifest:
Kubelet config:
Metrics Server logs:

E0224 04:18:59.393057       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0224 04:19:00.598542       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0224 04:19:07.205741       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-0-142-72.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-142-72.us-west-2.compute.internal (ip-10-0-142-72.): Get https://ip-10-0-142-72.:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup ip-10-0-142-72. on 172.20.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:ip-10-0-133-93.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-133-93.us-west-2.compute.internal (ip-10-0-133-93.): Get https://ip-10-0-133-93.:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup ip-10-0-133-93. on 172.20.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:ip-10-0-142-180.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-142-180.us-west-2.compute.internal (ip-10-0-142-180.): Get https://ip-10-0-142-180.:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup ip-10-0-142-180. on 172.20.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:ip-10-0-128-79.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-0-128-79.us-west-2.compute.internal (ip-10-0-128-79.): Get https://ip-10-0-128-79.:10250/stats/summary?only_cpu_and_memory=true: dial tcp: lookup ip-10-0-128-79. on 172.20.0.10:53: no such host]
E0224 04:19:19.840580       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0224 04:19:20.103035       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority

/king bug

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 24 (9 by maintainers)

Most upvoted comments

Just in case someone else has this issue on EKS, if you access your cluster through a role, make sure that you have set the username field in the mapping.

kubectl describe -n kube-system configmap aws-auth

  groups:
  - system:masters
  rolearn: arn:aws:iam::123456789123:role/kubernetes-devops
  username: devops:{{SessionName}}  # <--- Ensure this has been specified.

Source: https://aws.amazon.com/premiumsupport/knowledge-center/eks-metrics-server/

hahasheminejad on Aug 28, 2023

@nemo83 Confirmed! For EKS you just need to customize the Helm value args to be kubelet-preferred-address-types=InternalIP.

values.yaml

args:
- --kubelet-preferred-address-types=InternalIP

then

helm upgrade metrics-server stable/metrics-server -f values.yaml
# or directly without another yaml file
helm upgrade metrics-server stable/metrics-server --set "args[0]=--kubelet-preferred-address-types=InternalIP"

Reference

The Ultimate EKS Course

abdennour on Mar 24, 2020

I have EKS setup, for me helped this:

Also keep in mind that in Step 2 seems like you don’t need to use specific username, it can be blablabla or whatever:

      - rolearn: arn:aws:iam::123456789123:role/kubernetes-devops
        groups:
          - system:masters
        username: blablabla:{{SessionName}}

I also don’t use --kubelet-insecure-tls. Keep in minde that with --kubelet-insecure-tls you will get troubles in debug for prometheus-adapter as you will not be able to open metrics like described here for debug on you local machine when you do kubectl proxy --port=8001.

Thank you @abdennour and @hahasheminejad for your comments with solutions, for me it was require usage both of them.

Hronom on Aug 28, 2023

enable port 4443 in your eks node security group where source is the cluster security group.

praveenkumarKajla on Jul 25, 2022

@serathius I have been working with EKS since last few months, this is the only issue I faced so far, not sure if I am the right person to help

smitthakkar96 on Feb 23, 2020

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-api-admin
subjects:
- kind: User
  name: kubelet-api
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io

And kubelet-preferred-address-types=InternalIP will solve your problem

smitthakkar96 on Feb 22, 2020