minikube: HPA not working, unknown target CPU

Im trying to auto scale the kuber pods but not able to get the hpa cpu metrics . so its failing to auto scale.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080

Ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-tutorial-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-tutorial-cluster-ip
          servicePort: 80

Services.yaml

apiVersion: v1
kind: Service
metadata:
  name: kubernetes-tutorial-cluster-ip
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: kubernetes-tutorial-deployment
  type: ClusterIP

CMD output

D:\docker\kubernetes-tutorial>kubectl get all
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/kubernetes-tutorial-deployment-7b78fb65c7-5dvz2   1/1     Running   0          2d18h

NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                       ClusterIP   10.96.0.1        <none>        443/TCP   2d21h
service/kubernetes-tutorial-cluster-ip   ClusterIP   10.108.184.150   <none>        80/TCP    2d20h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubernetes-tutorial-deployment   1/1     1            1           2d20h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/kubernetes-tutorial-deployment-7b78fb65c7   1         1         1       2d18h

NAME                                                                 REFERENCE                                   TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   <unknown>/2%   1         4         1          122m

Here horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment showing target is UNKNOWN.

This is the error I got

D:\docker\kubernetes-tutorial>kubectl describe hpa kubernetes-tutorial-deployment
Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48 +0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Im new to the kubernetes . so I don’t know , how to proceed further.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (4 by maintainers)

Most upvoted comments

@nikhils5501 Thanks, this solved it for me. More specifically I added the metrics section to the config for service-pod.yml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.replicaCount }}
  maxReplicas: 16
  targetCPUUtilizationPercentage: 70
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 70

+25

highway900 on Nov 26, 2019

The deployment file should contain resource limits in order for the metrics-server to calculate the CPU utilization. https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ Edit your deployment.yaml and add the resource limits part in order to fix this.

To read more on this, refer to the autoscaler algorithm: Please note that if some of the pod’s containers do not have the relevant resource request set, CPU utilization for the pod will not be defined and the autoscaler will not take any action for that metric. See the algorithm details section below for more information about how the autoscaling algorithm works. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

ns0092 on Aug 23, 2019

Fix for Minikube v1.5.2 + K8S v1.16.2

Disable the metric-server addon and install from https://github.com/kubernetes-sigs/metrics-server

$ minikube addons disable metrics-server
$ kubectl create -f deploy/1.8+/
# edit metric-server deployment to add the flags
# args:
# - --kubelet-insecure-tls
# - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
$ kubectl edit deploy -n kube-system metrics-server

I think we can close the issue with that

cmarquezrusso on Dec 30, 2019

In case it’s not obvious how to fix unknown, what you most likely be missing is following:

kubectl set resources deployment nginx -c=nginx --limits=cpu=100m,memory=64Mi

ryuheechul on Jun 3, 2020

I think this issue had been resolved.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080 
EOF

and I confirmed HPA is working.

$ kubectl autoscale deployment kubernetes-tutorial-deployment --cpu-percent=50 --min=1 --max=4
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment autoscaled

$ kubectl get hpa
NAME                             REFERENCE                                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   1%/50%    1         4         1          22s

govargo on Mar 9, 2020

Seems like there is some kind of issue with certificates and/or IP, workaround is disabling those:

--kubelet-insecure-tls

--kubelet-preferred-address-types=InternalIP

Could be related to a broken DNS (somehow), or could be our broken InternalIP (vs. hostname) ?

afbjorklund on Aug 8, 2019

It’s unclear what the issue is with minikube is here? Are we providing a broken metrics-server?

tstromberg on Jul 16, 2019