kubernetes: Complete help for kubectl top command when heapster is not in kube-system namespace

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST

After switching heapster from kube-admin to monitoring namespace:

$ kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
$ kubectl top pod
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

Using --namespace does not work either:

$ kubectl top pod --namespace=monitoring
W0416 17:58:15.174555    1159 top_pod.go:190] Metrics not available for pod monitoring/grafana-2017641066-rlq7c, age: 26m33.174541702s
error: Metrics not available for pod monitoring/grafana-2017641066-rlq7c, age: 26m33.174541702s
$ kubectl top node --namespace=monitoring                                                                         [17:58:36]
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

Kubernetes version: 1.6.1 Environment:

  • Cloud provider or hardware configuration: bare-metal
  • OS : ubuntu 16.04
  • Kernel : Linux vps343998 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: kubeadm 1.6.1

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 46
  • Comments: 26 (4 by maintainers)

Commits related to this issue

Most upvoted comments

After investigation, it exists several options not documented in help command of kubectl top, as described in the source code here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/top_node.go#L53

Maybe add this flags to the help description ?

For the record, the options are:

  • --heapster-namespace: Namespace Heapster service is located in")
  • --heapster-service: Name of Heapster service
  • --heapster-scheme: Scheme (http or https) to connect to Heapster as
  • --heapster-port: Port name in service to use

For me, the problem is I didn’t install the metric server and installing metric server is worked for me:

$ git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git $ kubectl create -f kubernetes-metrics-server/

reference from : https://stackoverflow.com/questions/52860209/kubernetes-1-11-could-not-find-heapster-for-metrics

hope it can helps.

Hi, I have the same behavior even if heapster is on the kube-system namespaces. What i don’t understand it’s why even if it’s an deprecated project replace by metrics-server do we still need to find an heapster service on the cluster…

Regards,

I can confirm this starts to appear again in kops 1.11.0 even below args for metrics server command line.

--kubelet-insecure-tls and
--kubelet-preferred-address-types=InternalIP

it seems above works fine kops 1.10.0 tested in k8s versions below for kops 1.10.0 and 1.11.0 1.10.12 1.11.1 1.11.3 1.11.5 1.11.7

I think this can be closed, since kubectl top node --help provides the requested information.

When error: metrics not available yet happened to me it ended up being an error with the metrics deployment itself. I did a kubectl get pods -n kube-system, found the metrics pod and a kubectl describe pod... helped me diagnose why the deployment wouldn’t work.

I had to patch this default metrics-server deployment: https://raw.githubusercontent.com/kubernetes/kops/master/addons/metrics-server/v1.8.x.yaml with these changes:

containers:
        - name: metrics-server
          image: gcr.io/google_containers/metrics-server-amd64:v0.3.0
          imagePullPolicy: Always
          command:
            - /metrics-server
            - --kubelet-preferred-address-types=InternalIP
            # - --source=kubernetes.summary_api:''

And that got things working for me.

I got the same issue with Kubernetes 1.11 & latest Heapster.

$kubectl top pod --namespace=kube-system W0813 13:48:08.426536 27831 top_pod.go:263] Metrics not available for pod kube-system/calico-node-8d5d4, age: 38m1.426522624s error: Metrics not available for pod kube-system/calico-node-8d5d4, age: 38m1.426522624s

please make sure you’ve deployed metrics-server and have the metrics.k8s.io API correctly registered (check kubectl get apiservices)

@gdoctor these days, the only reason it should ask about Heapster is if the resource metrics API isn’t available. The kubectl top logic prefers using the resource metrics API, but will fall back to Heapster if it isn’t present (to give people time to upgrade clusters and clients), which means if neither are working, you’ll see an error about Heapster.

@s1113950 that kops manifest appears to be out of date/not actually tests. v0.3.0 changed how the options are specified, so it makes sense that you needed to make that change. Please file an issue with the kops repository.

For me, the problem is I didn’t install the metric server and installing metric server is worked for me:

$ git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git $ kubectl create -f kubernetes-metrics-server/

reference from : https://stackoverflow.com/questions/52860209/kubernetes-1-11-could-not-find-heapster-for-metrics

hope it can helps.

this was the only solution for me thanks

I know this --kubelet-insecure-tls will make it work. but is important to note that this is useful in test clusters with self-signed Kubelet serving certificates but is not recommended for production usage

Same problem using kubernetes v1.12.2 on premise usin kubespray to install