amazon-kinesis-scaling-utils: The kinesis autoscaler can't scale up by less than double

So a little while ago, after running into issues using a scale up percentage of less than 100%, I submitted this PR. My understanding was if I had an (abridged) config like:

"scaleUp": {
  "scaleThresholdPct": 75,
  "scaleAfterMins": 1,
  "scalePct": 115 
}

and I had a stream that currently had 100 shards, that the kinesis autoscaler would say “100 shards * 1.15, okay the stream will have 115 shards when I scale up”.

As far as I can tell from looking at the code though, that’s not actually the case, as this line of code indicates that the autoscaler interprets scalePct: 115 as “Add 115% the shard’s current capacity to it’s existing capacity”. This means that scalePct: 115 on a stream with 100 shards will actually scale the stream up to 215 shards.

The issue here isn’t that this is the behaviour, that’s totally fine; however, the config parser will throw an error if scaleUpPct is less than 100, meaning that any scale up operation must at least double the capacity of the stream.

I’m happy to go in an modify this in whatever way is necessary - change it so that we can use a scaleUpPct > 100, or change the scaling behaviour, but I’m not sure what the actual expected behaviour is - I’m hoping the maintainers can provide some clarity on this 😃

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 15 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Hey there - yes that was intentional. I’d rather we not scale down and leave the stream with ample capacity than over-scale and result in throttling. This could be added as a switch to the overall architecture, but I think it’s better to be conservative on scaling down - as you find elsewhere with cool-offs in EC2 etc.