metrics-server: Unable to get CPU for container discarding data: missing cpu usage metric

I added the following to the 1.8+ manifest and deploy it to EKS.

imagePullPolicy: Always
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP

Yet, I see the following errors:

I0123 02:04:30.824820       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0123 02:04:31.420786       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0123 02:04:31.448142       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2019/01/23 02:04:31 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/01/23 02:04:31 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0123 02:04:31.493567       1 serve.go:96] Serving securely on [::]:443
E0123 02:04:34.290945       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-1: no metrics known for pod
E0123 02:04:34.290964       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 02:05:04.301881       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-1: no metrics known for pod
E0123 02:05:04.301998       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 03:11:31.654059       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-0-1-40.ap-southeast-1.compute.internal: unable to get CPU for container "iconverse-nlp" in pod default/iconverse-nlp-1 on node "10.0.1.40", discarding data: missing cpu usage metric, unable to fully scrape metrics from source kubelet_summary:ip-10-0-0-168.ap-southeast-1.compute.internal: unable to get CPU for container "iconverse-converse" in pod default/iconverse-converse-0 on node "10.0.0.168", discarding data: missing cpu usage metric]
E0123 03:11:35.994020       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 03:12:06.018005       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 04:31:31.563306       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-10-0-2-172.ap-southeast-1.compute.internal: unable to get CPU for container "iconverse-nlp" in pod default/iconverse-nlp-0 on node "10.0.2.172", discarding data: missing cpu usage metric

$ k --kubeconfig=$KUBECONFIG get hpa
NAME                 REFERENCE                        TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
iconverse-converse   StatefulSet/iconverse-converse   <unknown>/50%   2         5         2          4h
$ k --kubeconfig=$KUBECONFIG describe hpa iconverse-converse
Error from server (NotFound): the server could not find the requested resource

This is my hpa manifest:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: myapplication
  namespace: default 
spec:
  maxReplicas: 5
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1 
    kind: StatefulSet
    name: myapplication
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 50

Any insight and advice is appreciated.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 16
Comments: 20 (3 by maintainers)

Most upvoted comments

Closing per Kubernetes issue triage policy

GitHub is not the right place for support requests. If you’re looking for help, check Stack Overflow and the troubleshooting guide. You can also post your question on the Kubernetes Slack or the Discuss Kubernetes forum. If the matter is security related, please disclose it privately via https://kubernetes.io/security/.

+25

serathius on Feb 7, 2020

maybe you didnt provide ca.crt and ca.key to the node of your cluster

+20

wang510918 on Jan 23, 2019

Same here in K8s 1.13.5

cjingtao-a01:metrics-server-0.3.2 cjingtao$ KUBECONFIG=/Users/cjingtao/Documents/GitHub/deploy/Server-End\ Complex/dev/Dev-CD.kubeconfig.yml kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

cjingtao-a01:metrics-server-0.3.2 cjingtao$ KUBECONFIG=/Users/cjingtao/Documents/GitHub/deploy/Server-End\ Complex/dev/Dev-CD.kubeconfig.yml kubectl logs metrics-server-68d85f76bb-8bb9p -n kube-system
I0505 02:06:44.758400       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2019/05/05 02:06:45 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/05/05 02:06:45 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0505 02:06:45.133211       1 serve.go:96] Serving securely on [::]:443
E0505 02:07:14.671030       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.je5voly0": no metrics known for node
E0505 02:07:14.671049       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.t2mzj8lv": no metrics known for node
E0505 02:07:14.671053       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.4q1e2xzl": no metrics known for node
E0505 02:07:14.671056       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.4iqiwkgz": no metrics known for node
E0505 02:07:14.671060       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.vrdnkjkv": no metrics known for node
E0505 02:07:14.671063       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.kkvwkpqv": no metrics known for node
E0505 02:07:14.671066       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.3wjheffw": no metrics known for node
E0505 02:07:14.671069       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.hxqlgl0o": no metrics known for node
E0505 02:07:14.671072       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.r85r2n3n": no metrics known for node
E0505 02:07:19.380532       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-launcher-1-6b4bd7d8cc-gl8v6: no metrics known for pod
E0505 02:07:19.380547       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-launcher-1-6b4bd7d8cc-s6bzs: no metrics known for pod
E0505 02:07:19.380551       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-launcher-1-6b4bd7d8cc-nxpvn: no metrics known for pod
E0505 02:07:19.380554       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-se-wrapper-1-64c978ccbc-q5btm: no metrics known for pod

cjingtao-a01:metrics-server-0.3.2 cjingtao$ KUBECONFIG=/Users/cjingtao/Documents/GitHub/deploy/Server-End\ Complex/dev/Dev-CD.kubeconfig.yml kubectl get pods metrics-server-68d85f76bb-8bb9p -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2019-05-05T02:06:41Z"
  generateName: metrics-server-68d85f76bb-
  labels:
    k8s-app: metrics-server
    pod-template-hash: 68d85f76bb
  name: metrics-server-68d85f76bb-8bb9p
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: metrics-server-68d85f76bb
    uid: 6c661bc4-6eda-11e9-a6de-020061cb48ef
  resourceVersion: "5736885"
  selfLink: /api/v1/namespaces/kube-system/pods/metrics-server-68d85f76bb-8bb9p
  uid: 6c6927a2-6eda-11e9-a6de-020061cb48ef
spec:
  containers:
  - command:
    - /metrics-server
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
    image: k8s.gcr.io/metrics-server-amd64:v0.3.1
    imagePullPolicy: Always
    name: metrics-server
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: tmp-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: metrics-server-token-nrdwl
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: kube-worker.t2mzj8lv
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: metrics-server
  serviceAccountName: metrics-server
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: tmp-dir
  - name: metrics-server-token-nrdwl
    secret:
      defaultMode: 420
      secretName: metrics-server-token-nrdwl
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:41Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:44Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:44Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:41Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://d49d30c912365bca3fb467f38a8e79a1c13c1f61d0a3edcc136d8ef6a5d7555e
    image: k8s.gcr.io/metrics-server-amd64:v0.3.1
    imageID: docker-pullable://k8s.gcr.io/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b
    lastState: {}
    name: metrics-server
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-05-05T02:06:44Z"
  hostIP: 10.92.101.22
  phase: Running
  podIP: 10.36.0.5
  qosClass: BestEffort
  startTime: "2019-05-05T02:06:41Z"

NKUCodingCat on May 5, 2019

There are 200 forks, does anyone know which (if any) of them fix metric server?

mabushey on Jan 29, 2019

I had the same problem

E1113 05:09:57.112222       1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:k8s-node-1: unable to get CPU for container "paas-micro-voucherwallet" in pod default/paas-micro-voucherwallet-6b9796f9cf-lm6qv on node "10.127.0.17", discarding data: missing cpu usage metric

kubectl version v1.15.0

metrics-server-amd64:v0.3.6

koy1619 on Nov 13, 2019

Thanks @zouyee. I changed from using “args” to “commands” and I don’t see the 401 Unauthorized now. However, kubectl logs -f metrics-server... -n kube-system still shows “no metrics known for pod”:

$ k logs -f metrics-server-68df9fbc9f-dgr8v -n kube-system
I0312 03:55:41.841800       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0312 03:55:42.433339       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0312 03:55:42.439873       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2019/03/12 03:55:42 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/03/12 03:55:42 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0312 03:55:42.488139       1 serve.go:96] Serving securely on [::]:443
E0312 03:55:46.554516       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-nlp-0: no metrics known for pod
E0312 03:55:46.554540       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-nlp-1: no metrics known for pod
  E0312 05:08:42.634201       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-192-168-84-18.ap-southeast-1.compute.internal: [unable to get CPU for container "iconverse-connector" in pod default/iconverse-connector-0 on node "192.168.84.18", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-connector-0 on node "192.168.84.18", discarding data: missing cpu usage metric], unable to fully scrape metrics from source kubelet_summary:ip-192-168-22-244.ap-southeast-1.compute.internal: [unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-converse-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-converse" in pod default/iconverse-converse-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-admin-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-admin" in pod default/iconverse-admin-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-ui" in pod default/iconverse-ui-0 on node "192.168.22.244", discarding data: missing cpu usage metric]]

khteh on Mar 14, 2019

same issue in EKS cluster with version 1.13

ghost on Sep 19, 2019