kubernetes: kubectl top node fails with heapster issue
We have a custom kube cluster. There is 1 Master and 2 Worker Node.
kubectl version is 1.5
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:52:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Referred to : https://github.com/kubernetes/kubernetes/issues/30818 similar issue.
heapster version is v1.3.0-beta.1 in the default namespace.
kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default heapster-2326584968-qwqmf 1/1 Running 0 22h
default monitoring-grafana-1918260480-njf8s 1/1 Running 0 22h
default monitoring-influxdb-201485718-mvkb5 1/1 Running 0 10d
kube-system calico-node-56cps 2/2 Running 0 10d
kube-system calico-node-6pcjx 2/2 Running 0 10d
kube-system calico-node-hqj3k 2/2 Running 0 10d
kube-system calico-policy-controller-261675450-64c8j 1/1 Running 0 10d
kube-system etcd-cni-kube-master-1 1/1 Running 0 10d
kube-system etcd-k8s-kube-master-1 1/1 Running 0 10d
kube-system kube-apiserver-kube-master-1 1/1 Running 0 10d
kube-system kube-controller-manager-kube-master-1 1/1 Running 1 10d
kube-system kube-dns-1.9-pbjb7 3/3 Running 0 10d
kube-system kube-proxy-4mzd6 1/1 Running 0 10d
kube-system kube-proxy-dqfmk 1/1 Running 0 10d
kube-system kube-scheduler-kube-master-1 1/1 Running 1 10d
kube-system kubernetes-dashboard-cb62k 1/1 Running 0 10d
cc @piosz
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 1
- Comments: 18 (8 by maintainers)
Having same issue as well. Im using 1.13 .
Since your heapster is not in
kube-system
namespace, trykubectl top node --heapster-namespace='default'
, please.FYI: #44540
I am also facing extact same issue mentioned by @ianjuma
I am using 1.10.1
Should we reopen this issue or create new one?
Appears to be linked to issue #56226 and #55238 ; where for aggregated services, pod/service networking is needed on the apiserver for this to work. Running kube-proxy and adding pod connectivity fixed this for me. Error 503 is also misleading here. Not sure if this is the apiserver/ heapster.
It can be because of client and server version mismatch @MilenkoM Please check the output of
and I also noticed this issue while using GKE 1.10.4 and 1.10.2. As for now switched back to GKE 1.9.7
@ompraash You should always run heapster service and deployment in kube-system namespace, I believe even dashboard will not be able to display metrics if you run it in different namespace If you run it in kube-system, you can also avoid necessity of passing --heapster-namespace as it defaults to kube-system
$ kubectl top node --heapster-namespace=kube-system NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-gce-us-central1-default-pool- 79m 7% 1685Mi 45%
gke-gce-us-central1-default-pool- 69m 6% 1680Mi 45%
gke-gce-us-central1-default-pool- 61m 6% 1869Mi 50%
$ kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
gke-gce-us-central1-default-pool- 68m 6% 1676Mi 45%
gke-gce-us-central1-default-pool- 62m 6% 1864Mi 50%
gke-gce-us-central1-default-pool- 77m 7% 1685Mi 45%
@suyogbarve 👍
@ompraash maybe you can try to deploy your heapster in
kube-system
namespace if you don’t want to upgrade.