aws-cdk: Throttling: Rate exceeded

We hit this issue regularly and it is getting really annoying 🤨

Last build 2 of 10 stacks failed with the “Throttling: Rate exceeded” error …a retrigger of the CICD pipeline will most likely succeed!

Just to add another voice to this This is affecting my team as well.

In particular we have several CDK apps which creates over 100 stacks each If more then one of these apps are deploying at once time they fail with the rate exceeded message and just exit failing our CI build with no apparent retries.

Can we please get some attention on this issue? My team is suffering from this.

Ran into something very similar as @calebpalmer, where I was creating ~20 lambdas with a custom log retention (logRetention prop of Function construct). Similarly, adding a few explicit dependencies remediated the issue.

In my case, CDK created a Custom::LogRetention resource for each function, and it failed with Rate exceeded errors in CloudFormation on the 16th resource and onwards.

Coincidentally there’s a non-adjustable Lambda quota limit (Rate of control plane API requests) of 15 API operations (per second?). Perhaps it’s related.

As this is clearly persisting, I am reopening the issue for further discussion

We love the CDK, but throttling from CloudFormation when using the CDK is still an issue for our team. We often have CDK builds/deploys running for different stages/stacks in our AWS Account at once and we run into this quite often.

We’re currently on CDK version 1.54.0

@Silverwolf90 that does not sound ideal at all and we should be providing a better experience natively.

bumping this up to a p1 as it’s affecting a lot of our users.

We are also seeing this issue, hoping we could have a way to configure the retries.

Is the reason for not having a configurable retry (via environment variables, config file) is because CDK is using the aws-sdk v2?

Hopefully, this could also be fixed in CDK v2

We also have the same issue in our team, the #8711 did not fix the issue for us. I would like to see a simple option to control the poll interval for the cdk cli command in order to avoid exceeding the rate.

I don’t know if this is related, but I’ve started seeing similar throttling errors in a single stack when trying to create an IAM role within a stack:

10/25 | 12:01:36 | CREATE_FAILED        | AWS::IAM::Role                 | SingletonLambda3f2d0f3dc42f4a18ab66a6ebeb8fa36f/ServiceRole (SingletonLambda3f2d0f3dc42f4a18ab66a6ebeb8fa36fServiceRoleDAA100A1) Rate exceeded (Service: AmazonIdentityManagement; Status Code: 400; Error Code: Throttling; Request ID: f4dd183c-5fd3-4a26-a6ce-4e1f34924fa7)
       new Role (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-kernel-P2u1xG\node_modules\@aws-cdk\aws-iam\lib\role.js:41:22)
       \_ new Function (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-kernel-P2u1xG\node_modules\@aws-cdk\aws-lambda\lib\function.js:61:35)
       \_ SingletonFunction.ensureLambda (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-kernel-P2u1xG\node_modules\@aws-cdk\aws-lambda\lib\singleton-lambda.js:58:16)
       \_ new SingletonFunction (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-kernel-P2u1xG\node_modules\@aws-cdk\aws-lambda\lib\singleton-lambda.js:19:36)
       \_ C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:7877:49
       \_ Kernel._wrapSandboxCode (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:8350:20)
       \_ Kernel._create (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:7877:26)
       \_ Kernel.create (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:7621:21)
       \_ KernelHost.processRequest (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:7411:28)
       \_ KernelHost.run (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:7349:14)
       \_ Immediate._onImmediate (C:\Users\JOHN~1.TIP\AppData\Local\Temp\jsii-java-runtime14499259584469591882\jsii-runtime.js:7352:37)
       \_ processImmediate (internal/timers.js:456:21)

picking this task up

Also seeing this issue. Like @clifflaschet, my problem seems related to introducing a log retention policy to existing stack with lambdas.

What about avoiding polling altogether while scaling to a large number of stacks in parallel - have a CDK service endpoint which CDK clients would subscribe to. Once a stack is finished deploying, the client will get (event-driven) notification and continue to the next stack.

I originally thought my rate limiting issue was from this but it actually ended up being rate limits reached between cloudformation itself and the services it was interacting with. For example I had a ton of independent lambda functions being created at the same time which caused rate limit errors between cloudformation and lambda. After adding some explicit dependencies between the lambdas it reduced the amount of them being created in parallel and eliminated the rate limiting issues. There were some other resources I had to do the same with like API Gateway models and methods. I’m mentioning this in case someone else in here might have came to the wrong conclusion like I did.

@RomainMuller @SomayaB @shivlaks Can you please reopen this issue? It clearly is not fixed yet. I’m using an updated version but it still happens the same way it did when I first opened this issue. You can also see a lot of people seeing the same error after this was closed.

There was an IAM issue overnight but it appears to be resolved or is in the process of resolving now

Hi @danielfariati, thanks for reporting this. We will update this issue when there is movement.

I’m getting this error in CDK v2

 ❌  test-platform-MyService failed: Throttling: Rate exceeded
    at Request.extractError (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/Users/cyber/dev/platform/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'Throttling',
  time: 2022-03-07T14:34:28.758Z,
  requestId: 'f3c002b8-f911-4ec2-a454-e3adcad8fd39',
  statusCode: 400,
  retryable: true
}
Rate exceeded

 ❌  test-platform-MyService failed: The test-platform-MyService stack failed to deploy.

Given that the error is retryable, maybe retry it before blowing up?

Please bump in priority. This issue is blocking us as well.

Same here 😦 I’m just trying to describe the cloudformation stacks and loop through them but keep encountering this error often. Throttling: Rate exceeded at Request.extractError at Request.callListeners at Request.emit at Request.emit at Request.transition at AcceptorStateMachine.runTo

I’m also seeing this. It’s driving me nuts. I have to comment stuff out and do multiple deploys to get my stack up which also means I can’t do a full deploy from a CI/CD system.

@richardhboyd

I’m still exploring the options, but some of the things we are considering include:

• make retries more configurable
• allow opting out of polling altogether
• handle rate throttled more gracefully (after exhausting retries and a better backoff, we might need to bail and just provide a link to the stack ARN and perhaps CloudFormation console link)

The downside of bailing on the stack monitoring is subsequent deploys will not be initiated by the CDK. i.e. if stack B had a dependency that required stack A to be deployed. We can’t start that deployment until A has completed. That would not be possible if we stopped monitoring.

This would affect wildcard deployments and any scenario where we can’t reason about the status of the stack without polling.

Handling rate limiting more gracefully is a precursor to attempting parallel deployments. Let me know if you had any additional thoughts, and I’ll work that in as I’m trying to prototype a proof of concept and test out the tradeoffs.

This is my understanding, so please don’t 🔥 me 😉

This issue has to do with CDK CLI being throttled because it hits the CloudFormation API too often and there is no way to override the defaults. This happens more often if you deploy multiple stacks in parallel.

The other rate limiting problem that folks are seeing is related to Custom resources in the stack itself. Primarily, with log retention. CloudWatch Logs has really low API limits (like 5 reqs/sec). You can fix this by using the logRetentionRetryOptions (docs). Lambda function construct has this option as well.

currently, we are running 11 deployments in parallel. we faced the issue the first time that Rate exceeded. day by day the parallel deployment count has been increasing.

any alternate solution is there is right now? because we don’t want to fail our pipeline for 1 2 deployments.

Thanks.

Just tried again updating 35 stacks in parallel and the issue still persists. Our end goal is to be able to deploy all stacks at once (~80, and growing… it was 27 when I first opened this ticket). In our case, the error is not on CloudFormation, like some users are reporting, but on CDK itself (probably calling CloudFormation to get the stack status). The stack on CloudFormation updates successfully, but CDK breaks (Rate Limit Exception), so the person running it lose track of what is happening. We are using CDK v2.

The error looks like this:

02:43:52   ❌  MY_STACK_NAME failed: Error: Rate exceeded
02:43:52      at prepareAndExecuteChangeSet (/usr/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:386:13)
02:43:52      at runMicrotasks (<anonymous>)
02:43:52      at processTicksAndRejections (internal/process/task_queues.js:95:5)
02:43:52      at deployStack2 (/usr/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:240:24)
02:43:52      at /usr/lib/node_modules/aws-cdk/lib/deploy.ts:39:11
02:43:52      at run (/usr/lib/node_modules/p-queue/dist/index.js:163:29)
02:43:52  
02:43:52   ❌ Deployment failed: Error: Stack Deployments Failed: Error: Rate exceeded
02:43:52      at deployStacks (/usr/lib/node_modules/aws-cdk/lib/deploy.ts:61:11)
02:43:52      at runMicrotasks (<anonymous>)
02:43:52      at processTicksAndRejections (internal/process/task_queues.js:95:5)
02:43:52      at CdkToolkit.deploy (/usr/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:312:7)
02:43:52      at initCommandLine (/usr/lib/node_modules/aws-cdk/lib/cli.ts:349:12)

Any news on this? The issue is tagged as p1, but it seems that nobody is looking into it. The issue is affecting us for more than 2 years now, and no workaround is possible (that I know of). 😢

What we’re currently doing is limiting 15 stacks in parallel, but this is becoming a huge problem, as our number of stacks is growing…

This has been plaguing our deploys for ages! Thank you for following up about #8257!

On Mon, Aug 7, 2023 at 12:24 AM Shane Argo @.***> wrote:

I cannot believe that the following issue has not been referenced from this one: #8257 https://github.com/aws/aws-cdk/issues/8257

The solution to this problem has been implemented. I had to use higher numbers for maxRetries and base (I used 20 and 1000ms respectively) than the user in this issue, but I managed to get my project to deploy.

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-cdk/issues/5637#issuecomment-1667329015, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOKSZBDY4WYYCOK4ZXHPDXUCJ4HANCNFSM4KCQZ2TQ . You are receiving this because you commented.Message ID: @.***>

I guess one way to solve this for good would be to create a utility API Gateway Websocket (the ones that API gateway supports natively), as part of the Bootstrap stack and subscribe to events on that one through the CDK CLI. That would help CDK drop the polling approach and would help the CLI get immediate feedback when a stack is deployed or fails to be deployed + a bonus - no throttling (since there are no longer any direct AWS API calls involved.

Side note: A third party library cdk-watch already does something similar under the hood. Talking about the CLI + Websocket API integration for realtime updates.