autoscaler: AWS: Can't scale up from 0

Possibly related: #1754

I recently added three new node groups to my cluster using AWS spot instances. I initially set the minSize on each of the three new groups to 0, but CA was refusing to scale them up from 0. If I go into the EC2 console and manually force the ASG minSize up to 1 then CA gets unstuck and will continue scaling the group up as new requests come in.

I’m attaching the following files:

ca_logs.txt :: At this point I had forced one of my ASGs to have a minSize of 1 and maxSize of 4. That group filled up so CA was unable to scale it up any further. At this point it should have been scaling up the other two node groups, but they still had minSize=0 and thus CA refused to scale them up.
ca_logs_after_setting_min.txt :: This is after manually forcing the two other ASGs to have minSize=1. At this point CA starts scaling them up as expected.
ca_pod.txt :: Full get pod -o yaml of my CA

Is it not supported to have minSize=0 on AWS?

I’m running CA v1.14.5.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 32 (16 by maintainers)

Commits related to this issue

docs: autoscaling: Add note about ASG tags for scaling up from 0 When the cluster-autoscaler adds a new node to a group, it grabs an existing node in the group and builds a "template" to launch a new... — committed to mgalgs/eksctl by mgalgs 5 years ago
docs: autoscaling: Add note about ASG tags for scaling up from 0 When the cluster-autoscaler adds a new node to a group, it grabs an existing node in the group and builds a "template" to launch a new... — committed to mgalgs/eksctl by mgalgs 5 years ago
Merge pull request #3453 from nilo19/bug/cherry-pick-3418-1.19 Cherry pick the bug fix in #2418 onto 1.19 — committed to kubernetes/autoscaler by k8s-ci-robot 4 years ago
Merge pull request #3452 from nilo19/bug/cherry-pick-3418-1-18 Cherry pick the bug fix in #2418 onto 1.18 — committed to kubernetes/autoscaler by k8s-ci-robot 4 years ago

Most upvoted comments

@mgalgs thanks for documenting the solution to this. I just ran into this and you saved my day! 💖

agconti on Jul 23, 2020

@mgalgs The difference between two is

scale from 0 - CA build template from ASG LaunchTemplate or LaunchConfiguration, it won’t know some of the kubernetes node labels. In order to properly build it, we need to add ASG tags and CA will convert tags to labels to construct the node template.
scale from 1 - Since node is already there, CA will use real node as a template.

Do you have tag in your ASG? check here https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#scaling-a-node-group-to-0

If you still have the issues, I will try to see anything wrong in 1.14.

Jeffwan on Oct 12, 2019

@faheem-cliqz I probably already fix the issue you meet. Please check this 58f3f23#diff-ade7b95627ea0dd6b6f4deee7f24fa7eR323-R331

We will have a release next week

Cool will get back to you with updates once you guys releases 😃

faheem-nadeem on Oct 12, 2019

@faheem-cliqz I probably already fix the issue you meet. Please check this https://github.com/kubernetes/autoscaler/commit/58f3f23fc63c6f26b4651820bb3b8f812ed7803b#diff-ade7b95627ea0dd6b6f4deee7f24fa7eR323-R331

We will have a release next week

Jeffwan on Oct 11, 2019

Okay so I am on CA v1.14.5 and I finally got it working - I had to upgrade eksctl from v0.5.2 to 0.6.0 - it scales from 0 now!

chnsh on Oct 4, 2019