amazon-eks-ami: Metrics Server not working

According to https://aws.amazon.com/blogs/opensource/horizontal-pod-autoscaling-eks/, Metrics Server should work. In my case, it is failing. I’m probably missing something but I’m not sure what that is.

What I did:

# Create a cluster with eksctl

eksctl create cluster \
    -n devops25 \
    -r us-west-2 \
    --kubeconfig cluster/kubecfg-eks \
    --node-type t2.small \
    --nodes 3 \
    --nodes-max 9 \
    --nodes-min 3

export KUBECONFIG=$PWD/cluster/kubecfg-eks

# install tiller

kubectl create \
    -f https://raw.githubusercontent.com/vfarcic/k8s-specs/master/helm/tiller-rbac.yml \
    --record --save-config

helm init --service-account tiller

# Install Metrics Server

helm install stable/metrics-server \
    --name metrics-server \
    --version 2.0.2 \
    --namespace metrics

What I got:

kubectl -n metrics logs -l app=metrics-server

I0925 22:39:59.871109       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0925 22:40:00.439451       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0925 22:40:00.447678       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2018/09/25 22:40:00 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/09/25 22:40:00 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0925 22:40:00.503815       1 serve.go:96] Serving securely on [::]:443

kubectl top nodes

Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

kubectl version --output=yaml

clientVersion:
  buildDate: 2018-09-10T11:44:36Z
  compiler: gc
  gitCommit: a4529464e4629c21224b3d52edfe0ea91b072862
  gitTreeState: clean
  gitVersion: v1.11.3
  goVersion: go1.11
  major: "1"
  minor: "11"
  platform: darwin/amd64
serverVersion:
  buildDate: 2018-05-28T20:13:43Z
  compiler: gc
  gitCommit: 2bba0127d85d5a46ab4b778548be28623b32d0b0
  gitTreeState: clean
  gitVersion: v1.10.3
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: linux/amd64

Console shows that the cluster uses platform version eks.2.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 15

Most upvoted comments

From what I have experienced in a test EKS.2 cluster it is not working if you have Private DNS disabled in your VPC. Even if you set “–kubelet-preferred-address-types” to InternalIP it won’t work:

Bam. Experienced this same problem and enabling DNS hostnames for the VPC fixes it.

If you are using Terraform to build your EKS cluster, add the following to your aws_vpc resource:

   enable_dns_hostnames = true

You can apply this change and don’t have to rebuild the VPC or cluster or anything. It will take a few minutes for the hostnames to start resolving, so be patient and watch the metrics-server log. When it stops scrolling with errors, kubectl top pod should work once more.

+18

chrissnell on Mar 7, 2019

So, apparently you need to also allow port 443 to the nodes from the cluster control pane. (https://github.com/kubernetes-incubator/metrics-server/issues/45). I’m testing this now and will report back.

JessieAMorris on Sep 28, 2018

EKS seem to not install by default the Kubernetes Metrics Server https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html

TiloGit on Aug 13, 2019

That worked for me. Adding port 443 to the allowed communication ports was all I needed to do.

JessieAMorris on Sep 29, 2018

From what I have experienced in a test EKS.2 cluster it is not working if you have Private DNS disabled in your VPC. Even if you set “–kubelet-preferred-address-types” to InternalIP it won’t work:

[restful] 2018/12/03 13:36:41 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
E1203 13:37:53.710664 1 reststorage.go:129] unable to fetch node metrics for node "ip-yyy.eu-west-1.compute.internal": no metrics known for node
E1203 13:37:53.710688 1 reststorage.go:129] unable to fetch node metrics for node "ip-yyy.eu-west-1.compute.internal": no metrics known for node
E1203 13:37:53.710695 1 reststorage.go:129] unable to fetch node metrics for node "ip-yyy.eu-west-1.compute.internal": no metrics known for node
E1203 13:38:15.309381 1 reststorage.go:144] unable to fetch pod metrics for pod xxx: no metrics known for pod
E1203 13:38:41.891378 1 manager.go:118] unable to save metrics: duplicate pod xxx received
E1203 13:38:45.316283 1 reststorage.go:144] unable to fetch pod metrics for pod xxx: no metrics known for pod

pdefreitas on Dec 3, 2018

Another note for anyone who finds this thread. I had strict control on both Inbound and Outbound rules for my masters. Make sure everything is symmetrical: node allows 443 inbound from master, and master allows 443 outbound to node. It took me a while to notice that my masters did not have outbound to 443 allowed. You may end up in the same boat if you follow the recommendations. https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html

fiatala on Apr 3, 2019

@VinayVanama , looks like we need to specify the resources example:

resources: requests: cpu: 200m

it works fine after specifying the above in the container section

jithinjose2004 on Oct 16, 2018

I got the security groups configured as per the above discussion: kubectl get --raw “/apis/metrics.k8s.io/v1beta1/nodes” works fine . But I still see “ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu on container” and also “unable to fetch pod metrics for pod” @vfarcic , could you please help me out here ?

jithinjose2004 on Oct 15, 2018