metrics-server: Error from server (ServiceUnavailable): the server is currently unable to handle the request

executing the command kubectl top nodes, seems it’s freezed

executing the command kubectl get --raw /apis/metrics.k8s.io/v1beta1 it returned Error from server (ServiceUnavailable): the server is currently unable to handle the request

view the logs from metrics-server

http: TLS handshake error from 192.168.133.64:51926:EOF

192.168.133.64 is the internal ip address of api-server

from api-server log

E1214 02:06:35.625504       1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1214 02:07:05.573958       1 available_controller.go:311] v1beta1.metrics.k8s.io failed with: Get https://10.109.179.236:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1214 02:07:05.658841       1 memcache.go:134] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

kubelet version is 1.12.3 metrics-server 0.3.1

i have another clusters-set with the same version and configuration, metrics-server works just fine

api-server parameters

 - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --advertise-address=10.100.1.2
    - --allow-privileged=true
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 17
Comments: 16 (2 by maintainers)

Commits related to this issue

Fixed metrics-server-76f8d9fc69-fknl8 0/1 CrashLoopBackOff - https://github.com/kubernetes-sigs/metrics-server/issues/188#issuecomment-450609945 — committed to x1wins/CW-OVP by x1wins 3 years ago
Fixed metrics-server-76f8d9fc69-fknl8 0/1 CrashLoopBackOff - https://github.com/kubernetes-sigs/metrics-server/issues/188#issuecomment-450609945 — committed to Simba830/cw_ovp by deleted user 3 years ago

Most upvoted comments

original

https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

added --kubelet-insecure-tls

https://github.com/x1wins/CW-OVP/blob/master/k8s-manifests/components.yaml

kubectl create -f https://raw.githubusercontent.com/x1wins/CW-OVP/master/k8s-manifests/components.yaml

+11

x1wins on Feb 24, 2021

try to edit the metrics-server-deployment.yaml file and add command parameters :

  containers:
    command:
    - /metrics-server
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
    volumeMounts:
    - name: tmp-dir
      mountPath: /tmp

+11

d0o0bz on Dec 31, 2018

you can add this command: - /metrics-server - --metric-resolution=30s - --requestheader-allowed-names=aggregator - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP it will work

Darren-wh on Jun 10, 2019

Adding hostNetwork: true in the spec fixed my issue !

hostNetwork:
  enabled: true

Duplicate of #157

anrajme on Feb 16, 2020

@aneesh121 you save 2 days. Thanks!

Anyone provision the cluster with kubeadm, please start by checking the answer https://github.com/kubernetes-sigs/metrics-server/issues/188#issuecomment-586673541

For EKS users, it requires only args: - --kubelet-preferred-address-types=InternalIP as I explained here

abdennour on Mar 24, 2020