serverless: Serverless Rate exceeded for Cloudformation API
Hi there! I have an issue connected with throttling on CloudFormation API.
Right now there is:
- 26 functions
- 3 dynamo tables, 1 sqs queue.
- sum: 160 cf resources
The issue occurs on the serverless deployment
ServerlessError: An error occurred:
BroadcastToClientsEventSourceMappingDynamodbChannelsMessagesTable - Rate exceeded for operation 'Rate exceeded (Service: Lambda, Status Code: 429, Request ID: b8b7796b-e9d7-4c84-9615-7d2a98c1c85b, Extended Request ID: null)
'..node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:125:33
From previous event:
at AwsDeploy.monitorStack (node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:28:12)
at node_modules/serverless/lib/plugins/aws/lib/updateStack.js:115:28
I’m pasting cloudformation error. Its always the same on sqs and dynamodb stream events mapping.
This is my investigation:
- CloudFormation resources limit ~160 - checked
- Resources names limit (64) - I renamed to resources to the shortnames & error still occurs randomly - checked Core issue always comes from EventSourceMapping resource creation. ie. (BroadcastToClientsEventSourceMappingDynamodbChannelsMessagesTable - Rate exceeded for operation 'Rate exceeded (Service: Lambda, Status Code: 429)
- Error occurs only on “new stack creation” - all resources are triggered to deploy and then cloudformation returns rate exceed response on EventSourceMapping resource - checked
- Deploying new stack needs to comment 50% of the functions in the resources files - checked
- Splitting plugin doesn’t help in type/group mode
serverless.yml
broadcastToClients:
handler: bin/broadcastToClients
package:
include:
- bin/broadcastToClients
events:
- stream:
type: dynamodb
batchSize: 1
enabled: true
startingPosition: LATEST
arn:
Fn::GetAtt:
- channelsMessagesTable
- StreamArn
channelBackup:
handler: bin/channelBackup
package:
include:
- bin/channelBackup
events:
- sqs:
exist: true
arn:
Fn::GetAtt:
- backupQueue
- Arn
deploy command
sls deploy --verbose --stage xyz
Node Version: 12.18.1
Framework Version: 1.73.1
Plugin Version: 3.6.14
SDK Version: 2.3.1
Components Version: 2.31.6
Installed version
1.73.1
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 9
- Comments: 39 (17 by maintainers)
I got an update from our TAM that the issue has been reported by a number of customers and is being investigated by the Lambda service team.
Of course they ar not, it states in the error message that they are coming from AWS.
I am reporting that because in my opinion Serverless should be able to work around such cases or tell user what to check or modify in order to deploy its stack.
Judging by the number of people raising this recently - I’m assuming there’s been some recent change to concurrent creations of event source mappings. I’ll see if I can get some info from our TAM.
@adrianbanasiak and @herebebogans , I recently contacted to AWS support team regarding the issue. here is the response i got from them.
"Thank you for contacting AWS Premium Support. My Name is XXXXX , and I will be glad to assist! I understand that a CFN stack is failing due to error message related to exceeding a specific API quota when creating an event source mapping for a Lambda function. Please correct me if I am wrong.
I went ahead and reviewed your last conversation with my colleague and can confirm that the way to remediate this issue is to use the “DependsOn” attribute [1] on the CFN resource. This way the API operations performed by CFN on the backend are serialized so that you do not incur in exceeding any API quotas.
Regarding the exact number for this API quota being breached, this information do not seems to be publicly available [2] at the moment so I have no visibility over that. Note that even by not knowing this exact quota limit, implementing the solution provided will remediate the issue you are facing.
Regarding how to determine how many times this specific API call “CreateEventSourceMapping” is being made by other CFN stacks. I would suggest to look at CloudTrail events and filter for that specific API call. You can set “Event Name” as “CreateEventSourceMapping” and you will see the invocations performed when deploying the CFN stack."
I resolved the issue by setting the serverless-dependson-plugin in serverless.yml file and added the version “serverless-dependson-plugin”: “^1.1.2”, in pacakge.json . I am no longer getting the rate exceeded errors now. here is the reference link for that https://www.npmjs.com/package/serverless-dependson-plugin.
Hope this helps…
Sure, I am waiting for AWS support response. Will follow up when I get something back.
I had feedback from our TAM that the rollout of the fix was completed to all lambda regions at end of September. The stacks where I was having this issue now deploy consistently!
For people that were having this issue I’d also suggest you also check
aws lambda list-event-source-mappingsand check for ESMs with “Problem : function not found” and then delete them withaws lambda delete-event-source-mapping.We had a number of these in our accounts from failed previous deployments and they were preventing deployment of the new stack.
I believe then, that we should add this in Framework. PR with that is welcome!
@aralmpa I too got the event source mappings already existed errors, to resolve those errors I did use "aws lambda delete-event-source-mapping --uuid ‘xxxxxxxxxxxxxxxx’ --region xxxxx " cli command to delete those mappings.
I am grateful for the information, but the depends on plugin is a poor solution. We need something better from AWS.
@swarnalatha935 thanks - I was not aware of that plugin.
Was working fine until last week and we are facing the exact same issue:
ProcessSfRecordsChangedEventSourceMappingSQSSfRecordsChangedQueue Rate exceeded for operation ‘Rate exceeded (Service: Lambda, Status Code: 429, Request ID: 1115cc57-412a-40c6-8207-a49f91761fcb, Extended Request ID: null)’.