kubernetes: Metrics Server problems

/kind bug Hi, I did a hpa autoscaling practice, and used the metrics server, but there is also no value about resources of pods, here is what happened:

  1. Create deployments
# kubectl create -f httpd-deployment.yml 
deployment.apps/httpd created

# cat httpd-deployment.yml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: httpd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:2.2.32
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
  1. Create service
# kubectl create -f httpd-service.yml
service/httpd-svc created

# cat httpd-service.yml 
apiVersion: v1
kind: Service
metadata:
  name: httpd-svc
spec:
  type: NodePort
  selector:
    run: httpd
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80
  1. Create hpa
# kubectl create -f hpa_httpd_test.yml 
horizontalpodautoscaler.autoscaling/hpa-httpd created

# cat hpa_httpd_test.yml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-httpd
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: httpd
  minReplicas: 1
  maxReplicas: 10
  #targetCPUUtilizationPercentage: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 200Mi
  1. Get hpa information
# kubectl get hpa
NAME        REFERENCE          TARGETS                          MINPODS   MAXPODS   REPLICAS   AGE
hpa-httpd   Deployment/httpd   <unknown>/200Mi, <unknown>/50%   1         10        1          1m
Problems: there is no value about the resources of my pod!
  1. Describe the hpa
# kubectl describe hpa hpa-httpd 
Name:                                                  hpa-httpd
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 21 Aug 2018 16:05:27 +0800
Reference:                                             Deployment/httpd
Metrics:                                               ( current / target )
  resource memory on pods:                             <unknown> / 200Mi
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource memory: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age              From                       Message
  ----     ------                        ----             ----                       -------
  Warning  FailedGetResourceMetric       9s (x8 over 3m)  horizontal-pod-autoscaler  unable to get metrics for resource memory: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  9s (x8 over 3m)  horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API
while the metrics server is running without errors.
# kubectl get pods -n kube-system 
NAME                                       READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-5psw7                   1/1       Running   0          6d
coredns-78fcdf6894-77qhn                   1/1       Running   0          6d
etcd-cluster-2-master                      1/1       Running   0          6d
kube-apiserver-cluster-2-master            1/1       Running   0          6d
kube-controller-manager-cluster-2-master   1/1       Running   0          6d
kube-flannel-ds-dtjpq                      1/1       Running   0          6d
kube-flannel-ds-lflpz                      1/1       Running   0          6d
kube-proxy-pcwkd                           1/1       Running   0          6d
kube-proxy-ttcrf                           1/1       Running   0          6d
kube-scheduler-cluster-2-master            1/1       Running   0          6d
metrics-server-56dc49db67-5c7qx            1/1       Running   0          1d
  1. my kubernetes version
# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

OS:

#  cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel:

Linux cluster-2-master 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

And I didn’t find the parameter “–horizontal-pod-autoscaler-use-rest-clients=false” in the file: “/etc/kubernetes/manifests/kube-controller-manager.yaml”. Also, I didn’t run the heapster.

Could you please give me some suggestions? Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

Il n ‘y a pas de question dans la version 1.11, mais dans les kubernets 1.13.1, j’ ai trouvé la question.

  Warning  FailedGetResourceMetric       5m29s (x3 over 5m59s)  horizontal-pod-autoscaler  unable to get metrics for resource memory: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  5m29s (x3 over 5m59s)  horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API

@KielChan Thanks,I have fixed it by steps below:

  1. Enable the Aggregation Layer.
  2. Modify the parameters in metrics-server-deployment.yaml
command:
        - /metrics-server
        - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
        - --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem

The file “ca.pem” is created when I enable the Aggregation Layer. And notice that we need to mount the volume where the file is. When I finished these steps, the metrics server worked normally.

It also works for me , do you try metrics-server version 0.3.1 ?

command: - /metrics-server - --metric-resolution=30s - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP

int 0.3.1 try this command

@Bessonov, Umm, yeah - I was trying to say - --kubelet-preferred-address-types=InternalIP works for me. I tried it today only - but on minikube 😃 . I referred this video - https://www.youtube.com/watch?v=uxuyPru3_Lc Check if it’s useful for you or not…

Usage of kubelet-insecure-tls or insecure=true doesn’t seems like a solution. Can you please open the issue again? @hechuan73