beanstalk-deploy: Failed deployments due to throttling

I’ve been getting intermittent failed deployments, apparently due to the API requests being throttled. The output is:

Created new application version d00348b in Beanstalk.
Starting deployment of version d00348b to environment ***
Deployment started...

03:05:18 INFO: Environment update is starting.
##[error]Deployment failed: Error: Status: 400. Code: Throttling, Message: Rate exceeded
##[error]Node run failed with exit code 2

Has anybody else come across this issue or have any suggestions on how to rectify it?

Sometimes occurs even when doing the first deployment of the day.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21

Most upvoted comments

Published v8 which has the throttletest branch merged in. Don’t log it as a warning anymore when getting a throttled requests, just log it normally. Closing this issue now, will re-open if there are more problems.

einaregilsson on Mar 18, 2020

@kceb That’s actually a great idea, and what I probably should have been doing from the start. I’ve uploaded a special branch to try it out, can you try using

uses: einaregilsson/beanstalk-deploy@throttletest

in your deployment script and see whether that works for you? It will allow up to 5 consecutive requests to fail with a throttle error before failing the deployment, and will log a warning if it gets a throttle error. If it does work, please let me see the logs to get an idea of how many times the requests were throttled.

einaregilsson on Mar 13, 2020

Ok, have published v7 which logs the number of API calls when getting the throttling error. If anyone who gets the throttling error uses it and gets the error, please post back here how many api calls were made. Right now I’m thinking maybe if you have a lot of instances it’s calls times instances, so that’s why it happens, but it’s just a theory for now.

einaregilsson on Mar 10, 2020

@dblarons Just published v6 https://github.com/einaregilsson/beanstalk-deploy/releases/tag/v6 which polls less for updates. Try it out and let me know whether it works.

einaregilsson on Feb 18, 2020

Two instances. We use immutable environment updates, which slows our deployment down a bit compared to rolling deploys.

As far as I can tell, it’s not necessarily just the frequency (5s) that’s the issue, but that the frequency is too much over the ~15-20 minutes it takes for EB to finish our immutable deployment. I’m thinking there’s a # of updates per X minutes cap, not just a per second cap.

In any event, we were previously using the command line eb deploy option for our deployment, and that worked fine, so maybe you can just use the same polling period they did?

dblarons on Feb 14, 2020