metrics-server: 3.0 wont work - reststorage.go:101] unable to fetch node metrics for node "k8sdev01": no metrics known for node "k8sdev01"

updated v0.2.1 to v0.3.0. on 0.2.1 was working

on v0.3.0 gives this error:

 E0925 16:47:40.840589       1 reststorage.go:101] unable to fetch node metrics for node "k8sdev01": no metrics known for node "k8sdev01"
E0925 16:47:40.840616       1 reststorage.go:101] unable to fetch node metrics for node "k8sdev02": no metrics known for node "k8sdev02"
E0925 16:47:40.840621       1 reststorage.go:101] unable to fetch node metrics for node "k8sdev03": no metrics known for node "k8sdev03"
E0925 16:47:40.874031       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"), x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")] 
E0925 16:46:55.894491       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8sdev02: unable to fetch metrics from Kubelet k8sdev02 (k8sdev02): Get https://k8sdev02:10250/stats/summary/: dial tcp: lookup k8sdev02 on 10.96.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:k8sdev03: unable to fetch metrics from Kubelet k8sdev03 (k8sdev03): Get https://k8sdev03:10250/stats/summary/: dial tcp: lookup k8sdev03 on 10.96.0.10:53: server misbehaving, unable to fully scrape metrics from source kubelet_summary:k8sdev01: unable to fetch metrics from Kubelet k8sdev01 (k8sdev01): Get https://k8sdev01:10250/stats/summary/: dial tcp: lookup k8sdev01 on 10.96.0.10:53: server misbehaving]

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 8
  • Comments: 25 (6 by maintainers)

Commits related to this issue

Most upvoted comments

I am facing the same issue with metrics server. I tried – command:

  • /metrics-server
  • –kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
  • –kubelet-insecure-tls

But metrics-server is looking by hostname only: E1009 05:37:26.888923 1 reststorage.go:129] unable to fetch node metrics for node “k8s-master.cluster.k8.local”: no metrics known for node E1009 05:37:26.889005 1 reststorage.go:129] unable to fetch node metrics for node “node03.cluster.k8.local”: no metrics known for node E1009 05:37:26.889022 1 reststorage.go:129] unable to fetch node metrics for node “node02.cluster.k8.local”: no metrics known for node

command:
- /metrics-server
- --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
- --kubelet-insecure-tls

works for me.

I solved this by adding --kubelet-insecure-tls and --kubelet-preferred-address-types=InternalIP flags.

If you installed metrics-server using Helm then you can apply this change by running kubectl edit deployment metrics-server -n <your-namespace>.

It should look like this:

screen shot 2019-03-05 at 1 10 03 am

I solved this by adding node ip to coredns: kubectl edit configmap coredns -n kube-system

apiVersion: v1 data: Corefile: | .:53 { errors health hosts { 192.168.199.100 master.qls.com 192.168.199.220 node01.qls.com 192.168.199.215 node02.qls.com fallthrough } kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 proxy . /etc/resolv.conf cache 30 loop reload loadbalance } kind: ConfigMap

I am facing the same issue with metrics server. I tried – command: - /metrics-server - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname - --kubelet-insecure-tls

But metrics-server is looking by hostname only: E1009 05:37:26.888923 1 reststorage.go:129] unable to fetch node metrics for node “k8s-master.cluster.k8.local”: no metrics known for node E1009 05:37:26.889005 1 reststorage.go:129] unable to fetch node metrics for node “node03.cluster.k8.local”: no metrics known for node E1009 05:37:26.889022 1 reststorage.go:129] unable to fetch node metrics for node “node02.cluster.k8.local”: no metrics known for node

I also encountered this problem. When the name of the node is changed to IP, it is working normally.

When the name of the node is the host name, metrics-server cannot resolve the host.

E1017 08:56:31.136817       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube-master: unable to fetch metrics from Kubelet kube-master (kube-master): Get http://kube-master:10255/stats/summary/: dial tcp: lookup kube-master on 10.254.0.2:53: no such host
E1017 08:57:01.047583       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube-master: unable to fetch metrics from Kubelet kube-master (kube-master): Get http://kube-master:10255/stats/summary/: dial tcp: lookup kube-master on 10.254.0.2:53: no such host
E1017 08:57:31.049286       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:kube-master: unable to fetch metrics from Kubelet kube-master (kube-master): Get http://kube-master:10255/stats/summary/: dial tcp: lookup kube-master on 10.254.0.2:53: no such host

no, sorry

 E0925 19:49:16.320673       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8sdev03: unable to fetch metrics from Kubelet k8sdev03 (k8sdev03): Get https://k8sdev03:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:k8sdev01: unable to fetch metrics from Kubelet k8sdev01 (k8sdev01): Get https://k8sdev01:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:k8sdev02: unable to fetch metrics from Kubelet k8sdev02 (k8sdev02): Get https://k8sdev02:10250/stats/summary/: x509: certificate signed by unknown authority]
E0925 19:49:16.454403       1 reststorage.go:129] unable to fetch node metrics for node "k8sdev02": no metrics known for node
E0925 19:49:16.454429       1 reststorage.go:129] unable to fetch node metrics for node "k8sdev03": no metrics known for node
E0925 19:49:16.454433       1 reststorage.go:129] unable to fetch node metrics for node "k8sdev01": no metrics known for node