kubernetes: When increase cpu load HorizontalPodAutoscaler can't auto scale pod

Description of problem:
When increase cpu load, the pod can't auto scale, hpa can't works as expect.

Version-Release number of selected component (if applicable):
[root@ip-172-18-12-2 horizontal-pod-autoscaling]# kubectl version
Client Version: version.Info{Major:"1", Minor:"2+", GitVersion:"v1.2.0-alpha.2.90+360d0e0a7e92dc", GitCommit:"360d0e0a7e92dcc4d00b4c555e166a2a71a7e713", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2+", GitVersion:"v1.2.0-alpha.2.90+360d0e0a7e92dc", GitCommit:"360d0e0a7e92dcc4d00b4c555e166a2a71a7e713", GitTreeState:"clean"}

How reproducible:
Always.

Steps to Reproduce:
1. Enable experimental for kube-apiserver
1). For "kube-apiserver"  add  --runtime-config=extensions/v1beta1=true

2. Run & expose php-apache server
$ kubectl create -f rc-php-apache.yaml
$ kubectl expose rc php-apache --port=80
$ kubectl get pods
$ kubectl get svc
$ curl http://$php-server-ip

3. Create horizontal pod autoscaler
$ kubectl create -f hpa-php-apache.yaml
$ kubectl get hpa

4. Increase cpu load and check rc
$ while true; do curl http://$svcip:port; done
$ kubectl get hpa
$ kubectl get rc
$ kubectl get pod

Actual results:
4. When increase cpu load, the pod can't auto scale.
[root@ip-172-18-12-2 kubernetes]# kubectl get rc
CONTROLLER   CONTAINER(S)   IMAGE(S)                            SELECTOR         REPLICAS   AGE
php-apache   php-apache     horizontal-pod-autoscaling:latest   run=php-apache   1          58m
[root@ip-172-18-12-2 kubernetes]# kubectl get hpa
NAME         REFERENCE                                   TARGET    CURRENT     MINPODS   MAXPODS   AGE
php-apache   ReplicationController/default/php-apache/   1%        <waiting>   1         10        46m
[root@ip-172-18-12-2 kubernetes]# kubectl get pod
NAME               READY     STATUS    RESTARTS   AGE
php-apache-wuwzo   1/1       Running   0          58m

Expected results:
4. When increase cpu load, the pod should auto scale.

Additional info:
1.$ cat hpa-php-apache.yaml
apiVersion: extensions/v1beta1
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: default
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleRef:
    kind: ReplicationController
    name: php-apache
    namespace: default
  cpuUtilization:
    targetPercentage: 1
2.$ cat rc-php-apache.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    run: php-apache
  name: php-apache
  namespace: default
spec:
  replicas: 1
  selector:
    run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - image: horizontal-pod-autoscaling:latest
        imagePullPolicy: IfNotPresent
        name: php-apache
        resources:
          requests:
            cpu: 200m
      restartPolicy: Always
3. Image "horizontal-pod-autoscaling:latest" is build from 
https://github.com/kubernetes/kubernetes/tree/master/examples/extensions/horizontal-pod-autoscaling/image
4. The php-apache cpu usage can refer:
 http://fpaste.org/281762/44541276/

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 18 (11 by maintainers)

Most upvoted comments

@jszczepkowski For my env is local all-in-one. I setup the env just run command “hack/local-up-cluster.sh”. Then create heapster by json “cluster/addons/cluster-monitoring/standalone/heapster-controller.yaml” and hpa. Because kubelet listen on localhost 127.0.0.1 and heapster in container connect to 127.0.0.1, it will never connect to the master. At first, I don’t know how hpa works with heapster. Honestly after read the code, I know hpa get heapster from “kube-system” namesapce, it’s hard code and no doc describe this. So I make this mistake. I resolve this problem by create heapster in correctly namespace, change kube default listen ip, configure it in heapster yaml with "- --source=kubernetes:‘http://$kube-apiserver’ ".