autoscaler: Autoscaler fails to scale up nodes with pending pods

We are currently running a cluster-autoscaler in AWS, on a kops cluster. We had autoscaling enabled for an ASG where each node would have a taint and a label applied. We would then only schedule pods onto it that had a toleration for that taint.

Our setup was working correctly, until a few days ago, when scale up stopped happening (scale downs were working correctly). Most of the logs we saw were like this:

16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.493718       1 utils.go:130] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-26tbkn marked as unschedulable can be scheduled on ip-10-79-150-148.ec2.internal. Ignoring in scale up.
16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.494043       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-24mkp5 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.494225       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2xqhmb marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.494391       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2lb9tj marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.494558       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2grt2g marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.494747       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2pkg4w marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.494 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.494908       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2l5dk8 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.495 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.495072       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-22wjf9 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.495 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.495244       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2kv4xp marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.495 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.495404       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-25q2bq marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.495 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.495565       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2nqk9j marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.495 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.495733       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-26v2sv marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.496 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.495904       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2dd65s marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.496 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.496062       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2252x4 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.496 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.496226       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2znsb4 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.496 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.496390       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2d6xbs marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.496 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.496553       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-24b289 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.496 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.496723       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-29b6sc marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.497 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.496923       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2j5bkz marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.497 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.497095       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-28rfln marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.497 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.497295       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2pbfxx marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.497 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.497470       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2g46tk marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.497 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.497668       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-22fg6q marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.590 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.590031       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2xcdsf marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.690 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.689812       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2zqwmh marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.690 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.690170       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2dbksl marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.690 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.690435       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2z9lv9 marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.690 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.690673       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-265h6w marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.691 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.690949       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2l8cvk marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.
16:25:53.691 kops-k8s-cluster.services.opendoor.com I0703 23:25:53.691222       1 utils.go:125] Pod spark-worker-test-pyspark-notebook-jupyter-signals-spark-2l2klb marked as unschedulable can be scheduled (based on simulation run for other pod owned by the same controller). Ignoring in scale up.

It is unclear why autoscaler thinks these pods are schedulable, since when we disabled autoscaling and manually increase node count those pods were scheduled successfully.

We are running kubernetes 1.9.9 and the autoscaler with the following flags:

./cluster-autoscaler --cloud-provider=aws --namespace=kube-system --nodes=1:50:honeycomb-worker.kops-k8s-cluster.services.opendoor.com --v=4 --alsologtostderr=true

It is running using this docker image k8s.gcr.io/cluster-autoscaler:v1.1.2. Any help in how to fix this would be greatly appreciated since we would love to re-enable cluster autoscaler.

Context

We are using kubernetes and cluster autoscaler to schedule periodically our cronjobs through the k8s API. This means that we create and delete hundreds of pods during ~30min period. Some of these pods are Spark pods, which might require large amounts of memory (~30Gb plus a couple of cores).

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 32 (12 by maintainers)

Most upvoted comments

The message “Pod ABC marked as unschedulable can be scheduled on ip-1-2-3-4.internal. Ignoring in scale up.” means there is inconsistency in what CA thinks and what scheduler thinks about cluster. Basically CA believes that pod ABC should have been scheduled on ip-1-2-3-4.internal. But for some reason the scheduler does not schedule it there. This is unexpected situation, yet possible if for example cluster uses non-standard scheduler, while CA simulates behavior of standard one. The safe approach is not to add extra nodes in such situation as it is hint that CA cannot simulate scheduler operation correctly in given cluster.

With all that said typically the message means that actually we triggered bug in scheduler which manifests in its internal cache inconsistency. With inconsistent cache scheduler believes the pods ABS is not schedulable anywhere despite the fact that there is place for it on ip-1-2-3-4.internal.

losipiuk on Nov 15, 2018

Defintiely haven’t seen any problems on 1.11.7

tcolgate on Mar 28, 2019

We have confirmed that 1.11.7 works for us.

On Wed, Feb 27, 2019, 8:23 AM Joe Lodato notifications@github.com wrote:

@tcolgate https://github.com/tcolgate can you confirm if it’s fixed for you in 1.11.7?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kubernetes/autoscaler/issues/1049#issuecomment-467929208, or mute the thread https://github.com/notifications/unsubscribe-auth/AZe3KOwHnVkpRNpEHu5oqKfkZ7QdQHn4ks5vRrEFgaJpZM4VElbS .

mlarrousse on Feb 28, 2019