kubernetes: hpa problem on k8s 1.9.0

i found a solution in my case: kube-controller-manager’s parameter --horizontal-pod-autoscaler-use-rest-clients in k8s 1.9.0 default value is true , while in k8s 1.8.x is false change it to false and it works.

same problem on version 1.10.0

With --horizontal-pod-autoscaler-use-rest-clients=true HPA uses new resource metrics API instead of old way of getting metrics. Setting it to false as you did works, but probably the correct long term solution is to run metrics server as part of your cluster set up. This is documented here: https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/

@MaciekPytel

Hi, I used the metrics server, but there is also no value about resource of pod, here is what happened:

1. Create deployments

# kubectl create -f httpd-deployment.yml 
deployment.apps/httpd created

# cat httpd-deployment.yml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: httpd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:2.2.32
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

1. Create service

# kubectl create -f httpd-service.yml
service/httpd-svc created

# cat httpd-service.yml 
apiVersion: v1
kind: Service
metadata:
  name: httpd-svc
spec:
  type: NodePort
  selector:
    run: httpd
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80

1. Create hpa

# kubectl create -f hpa_httpd_test.yml 
horizontalpodautoscaler.autoscaling/hpa-httpd created

# cat hpa_httpd_test.yml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-httpd
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: httpd
  minReplicas: 1
  maxReplicas: 10
  #targetCPUUtilizationPercentage: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 200Mi

1. Get hpa information

# kubectl get hpa
NAME        REFERENCE          TARGETS                          MINPODS   MAXPODS   REPLICAS   AGE
hpa-httpd   Deployment/httpd   <unknown>/200Mi, <unknown>/50%   1         10        1          1m

there is no value about the resource of my pod.

1. Describe the hpa

# kubectl describe hpa hpa-httpd 
Name:                                                  hpa-httpd
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 21 Aug 2018 16:05:27 +0800
Reference:                                             Deployment/httpd
Metrics:                                               ( current / target )
  resource memory on pods:                             <unknown> / 200Mi
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource memory: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age              From                       Message
  ----     ------                        ----             ----                       -------
  Warning  FailedGetResourceMetric       9s (x8 over 3m)  horizontal-pod-autoscaler  unable to get metrics for resource memory: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  9s (x8 over 3m)  horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API

while the metrics server is running without errors.

# kubectl get pods -n kube-system 
NAME                                       READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-5psw7                   1/1       Running   0          6d
coredns-78fcdf6894-77qhn                   1/1       Running   0          6d
etcd-cluster-2-master                      1/1       Running   0          6d
kube-apiserver-cluster-2-master            1/1       Running   0          6d
kube-controller-manager-cluster-2-master   1/1       Running   0          6d
kube-flannel-ds-dtjpq                      1/1       Running   0          6d
kube-flannel-ds-lflpz                      1/1       Running   0          6d
kube-proxy-pcwkd                           1/1       Running   0          6d
kube-proxy-ttcrf                           1/1       Running   0          6d
kube-scheduler-cluster-2-master            1/1       Running   0          6d
metrics-server-56dc49db67-5c7qx            1/1       Running   0          1d

1. my kubernetes version

# kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

OS:

#  cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel:

Linux cluster-2-master 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

And I didn’t find the parameter “–horizontal-pod-autoscaler-use-rest-clients=false” in the file: “/etc/kubernetes/manifests/kube-controller-manager.yaml”. Also, I didn’t run the heapster.

Could you please give me some suggestions? Thanks!

Right, so this is why it isn’t working. How to fix it will likely depend on how you set up your cluster (ie. kops / kube-up.sh / …). You should probably ask the maintainers of that tool (I only use GCP/GKE and it metrics-server works there out of the box).

For information (because I got a similar issue), here is the solution for a kubernetes cluster created using kops:

• for cpu or memory metrics: https://github.com/kubernetes/kops/blob/master/docs/horizontal_pod_autoscaling.md#support-for-multiple-metrics (!! Do not forget to install the metrics server addon (aka RTFM) !!)
• for custom metrics: https://github.com/kubernetes/kops/blob/master/docs/horizontal_pod_autoscaling.md#support-for-custom-metrics

To be added in cluster config (via kops edit cluster <clustername>) and applied using kops commands (kops update cluster <clustername> --yes + kops rolling-update cluster --name <clustername> --yes)

On done, you can then play with autoscaling on CPU / memory / custom metrics (well as soon as you have installed the required components to get the custom metrics API, RTFM)

I had somewhat the same problem. This solution worked for me: https://stackoverflow.com/q/54106725/2291510

Is there a way to set this property on Docker for mac?

@amirzaidi2002

if you using minikube, there are manifest files(.yaml) for kube-controller-manager

/etc/kubernetes/manifests/kube-controller-manager.yaml you can add new lines and config infos. parameter --horizontal-pod-autoscaler-use-rest-clients=false

That’s strange. How are you deploying your cluster? In 1.10 HPA all the RBAC should be set up by default, though that probably also depends on how the cluster is deployed. Also RBAC related errors in my experience have more explicit errors.

My guess would be the problem is either with missing RBAC or certs for aggregated API server.

Some things I would check to debug: kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods should print json with metrics for your pods (assuming you have any pods running in default namespace). If you get it it means metrics-server is running as expected. If not check if you have extension-apiserver-authentication configmap in kube-system namespace - missing that would probably mean the problem is with apiserver certs.

/kubectl get clusterrole system:controller:horizontal-pod-autoscaler -o yaml should return a yaml containing:

...
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  verbs:
  - list
...

If it’s not there you may need to add it.

If it’s none of the above you can try to look into kube-apiserver and metrics-server logs or open an issue against metrics-server repo (or the repo of your deployment tool).

@gjmzj how did you set the parameter --horizontal-pod-autoscaler-use-rest-clients=false. Please help. I am having a same problem and running on Kubernetes 1.9 (minikube)

it works well on k8s 1.8.6