kubernetes: HPA not working, unknown target CPU

Im trying to auto scale the kuber pods but not able to get the hpa cpu metrics . so its failing to auto scale.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080

Ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-tutorial-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-tutorial-cluster-ip
          servicePort: 80

Services.yaml

apiVersion: v1
kind: Service
metadata:
  name: kubernetes-tutorial-cluster-ip
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: kubernetes-tutorial-deployment
  type: ClusterIP

CMD output

D:\docker\kubernetes-tutorial>kubectl get all
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/kubernetes-tutorial-deployment-7b78fb65c7-5dvz2   1/1     Running   0          2d18h

NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                       ClusterIP   10.96.0.1        <none>        443/TCP   2d21h
service/kubernetes-tutorial-cluster-ip   ClusterIP   10.108.184.150   <none>        80/TCP    2d20h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubernetes-tutorial-deployment   1/1     1            1           2d20h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/kubernetes-tutorial-deployment-7b78fb65c7   1         1         1       2d18h

NAME                                                                 REFERENCE                                   TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   <unknown>/2%   1         4         1          122m

Here horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment showing target is UNKNOWN.

kubectl describe will be

This is the error I got

D:\docker\kubernetes-tutorial>kubectl describe hpa kubernetes-tutorial-deployment
Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48 +0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Im new to the kubernetes . I checked in stack overflow.but solutions not worked for me…so I don’t know , how to proceed further.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 18 (5 by maintainers)

Most upvoted comments

same issue here, any solution for this issue?

syedzac on Mar 12, 2020

Seeing a similar issue, but a different error: ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu

I am explicitly setting cpu/mem requests on all Deployments across the board in my cluster using an in-house Helm chart. Literally everything has explicit resource requests. Randomly, many hpa objects complain with this error.

I have verified that all releases have requests attached to their Deployments. We are using metrics-server, not heapster. I am using the latest version of metrics-server as well as the latest Helm chart.

We are using autoscaling/v2beta2 and most of the Deployments are using targetCPUUtilizationPercentage: foo for their requests. We are attempting to switch to the new format:

  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75

Switching to the newer format has mixed results - sometimes it works fine, other times I see the error complaining about missing requests.

This is a big issue. Any thoughts on why I’m just seeing this randomly?

echoboomer on Jun 20, 2019