metrics-server: unable to fully collect metrics

use k8s v1.11.2 && metrics-server v0.3.0

Did the following:

Add apiserver args like this: KUBE_API_ARGS="--authorization-mode=Node,RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kube-proxy.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-proxy-key.pem"
Add kubelet args like this: KUBELET_ARGS="--network-plugin=cni --cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --authentication-token-webhook=true --read-only-port=10255"
Ran kubectl create -f deploy/1.8+/
look metrics-server logs;

then metrics-server logs show this:

unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:192.168.66.66: unable to fetch metrics from Kubelet 192.168.66.66 (192.168.66.66): Get https://192.168.66.66:10250/stats/summary/: x509: cannot validate certificate for 192.168.66.66 because it doesn't contain any IP SANs, unable to fully scrape metrics from source kubelet_summary:192.168.66.64: unable to fetch metrics from Kubelet 192.168.66.64 (192.168.66.64): Get https://192.168.66.64:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.220.50: unable to fetch metrics from Kubelet 192.168.220.50 (192.168.220.50): Get https://192.168.220.50:10250/stats/summary/: x509: cannot validate certificate for 192.168.220.50 because it doesn't contain any IP SANs]

when I use metrics-server v0.2.1,add metrcs-server args like this:

command:
   - /metrics-server
   - --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true

It seems to work,but when i use v0.3.0 and add args --source it’s says Error: unknown flag: --source, So I used the help instruction to find the corresponding parameter, but I didn’t find it. So, how can I fix this problem?

This is my first time to mention issue, which is not so good. Sorry.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 32 (11 by maintainers)

Most upvoted comments

@vvkkhjt try this flag --kubelet-insecure-tls

MIBc on Sep 3, 2018

FIxed for me on EKS helm install stable/metrics-server --name metrics-server --namespace metrics --set args[0]=–kubelet-insecure-tls,args[1]=–kubelet-preferred-address-types=InternalIP

urvineet on Jul 5, 2019

Hi all,

k8s: v1.11 metrics-server: 0.3.1

I am not using TLS, However, I fall into similar problem(not being able to fully collect metrics). The logs of metrics-server show that it receives a - Raw (json) response - from kubelet, but i can’t retrieve the metrics via k top node/pod.

After receiving the kubelet’s sumary(json) there is such a line: ScrapeMetrics: time: 59.513576ms, nodes: 0, pods: 0

Another message says it cannot parse some init containers. Is this the reason it is failing? unable to get a valid timestamp for metric point for container

Best!

E0924 12:45:08.398855 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8s1: unable to get a valid timestamp for metric point for container “install-cni” in pod kube-system/kube-router-2l6f5 on node “10.36.10.1”, discarding data: no non-zero timestamp on either CPU or memory, unable to fully scrape metrics from source kubelet_summary:k8s4: unable to get a valid timestamp for metric point for container “install-cni” in pod kube-system/kube-router-6zvxd on node “10.36.10.4”, discarding data: no non-zero timestamp on either CPU or memory, unable to fully scrape metrics from source kubelet_summary:k8s5: [unable to get a valid timestamp for metric point for container “compile” in pod default/hello-6786bd6b44-t6z5z on node “10.36.10.5”, discarding data: no non-zero timestamp on either CPU or memory, unable to get a valid timestamp for metric point for container “prepare” in pod default/hello-6786bd6b44-t6z5z on node “10.36.10.5”, discarding data: no non-zero timestamp on either CPU or memory, unable to get a valid timestamp for metric point for container “install-cni” in pod kube-system/kube-router-gntmq on node “10.36.10.5”, discarding data: no non-zero timestamp on either CPU or memory]]

brunovlucena on Sep 24, 2018

@DirectXMan12 but the other services are working good。。when i use helm install heapster,it works good either。

helm install stable/heapster --name heapster --namespace kube-system --set "rbac.create=true,command[0]=/heapster,command[1]=--source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true" --debug

then

k top node

NAME             CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
192.168.220.50   254m         3%        3606Mi          11%       
192.168.66.64    226m         5%        2096Mi          57%       
192.168.66.66    99m          2%        1388Mi          62%

vvkkhjt on Sep 7, 2018