kubernetes: kubectl top node fails with heapster issue

We have a custom kube cluster. There is 1 Master and 2 Worker Node.

kubectl version is 1.5

Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:52:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Referred to : https://github.com/kubernetes/kubernetes/issues/30818 similar issue.

heapster version is v1.3.0-beta.1 in the default namespace.

kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
default       heapster-2326584968-qwqmf                  1/1       Running   0          22h
default       monitoring-grafana-1918260480-njf8s        1/1       Running   0          22h
default       monitoring-influxdb-201485718-mvkb5        1/1       Running   0          10d
kube-system   calico-node-56cps                          2/2       Running   0          10d
kube-system   calico-node-6pcjx                          2/2       Running   0          10d
kube-system   calico-node-hqj3k                          2/2       Running   0          10d
kube-system   calico-policy-controller-261675450-64c8j   1/1       Running   0          10d
kube-system   etcd-cni-kube-master-1                     1/1       Running   0          10d
kube-system   etcd-k8s-kube-master-1                     1/1       Running   0          10d
kube-system   kube-apiserver-kube-master-1               1/1       Running   0          10d
kube-system   kube-controller-manager-kube-master-1      1/1       Running   1          10d
kube-system   kube-dns-1.9-pbjb7                         3/3       Running   0          10d
kube-system   kube-proxy-4mzd6                           1/1       Running   0          10d
kube-system   kube-proxy-dqfmk                           1/1       Running   0          10d
kube-system   kube-scheduler-kube-master-1               1/1       Running   1          10d
kube-system   kubernetes-dashboard-cb62k                 1/1       Running   0          10d

cc @piosz

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 18 (8 by maintainers)

Most upvoted comments

Having same issue as well. Im using 1.13 .

$ kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
$

azmidotmy on Mar 9, 2019

Since your heapster is not in kube-system namespace, try kubectl top node --heapster-namespace='default', please.

FYI: #44540

xiangpengzhao on Jun 3, 2017

I am also facing extact same issue mentioned by @ianjuma

kubectl top node --heapster-namespace=kube-system
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get services http:heapster:)

I am using 1.10.1

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Should we reopen this issue or create new one?

nsidhaye on Oct 15, 2018

Appears to be linked to issue #56226 and #55238 ; where for aggregated services, pod/service networking is needed on the apiserver for this to work. Running kube-proxy and adding pod connectivity fixed this for me. Error 503 is also misleading here. Not sure if this is the apiserver/ heapster.

ianjuma on Aug 8, 2018

It can be because of client and server version mismatch @MilenkoM Please check the output of

kubectl version

and I also noticed this issue while using GKE 1.10.4 and 1.10.2. As for now switched back to GKE 1.9.7

bfarayev on Jun 21, 2018

@ompraash You should always run heapster service and deployment in kube-system namespace, I believe even dashboard will not be able to display metrics if you run it in different namespace If you run it in kube-system, you can also avoid necessity of passing --heapster-namespace as it defaults to kube-system

$ kubectl top node --heapster-namespace=kube-system NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-gce-us-central1-default-pool- 79m 7% 1685Mi 45%
gke-gce-us-central1-default-pool- 69m 6% 1680Mi 45%
gke-gce-us-central1-default-pool- 61m 6% 1869Mi 50%
$ kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-gce-us-central1-default-pool- 68m 6% 1676Mi 45%
gke-gce-us-central1-default-pool- 62m 6% 1864Mi 50%
gke-gce-us-central1-default-pool- 77m 7% 1685Mi 45%

suyogbarve on Jun 3, 2017

@suyogbarve 👍

@ompraash maybe you can try to deploy your heapster in kube-system namespace if you don’t want to upgrade.

xiangpengzhao on Jun 3, 2017