aws-sdk-js: Cloudfront Invalidation rate limit issues
Confirm by changing [ ] to [x] below to ensure that it’s a bug:
- I’ve gone through Developer Guide and API reference
- I’ve checked AWS Forums and StackOverflow for answers
- I’ve searched for previous similar issues and didn’t find any solution
- This is an issue with version 2.x of the SDK
Describe the question We have an application that is using S3 and cloudfront. we have define a process which will create or replace new object/files in S3 bucket which will consume in our mobile app with cloudfront URL.
Process - Once we get a file in S3 bucket, we triggered lambda function that execute invalidation for specific S3 object (without using * wildcard character).
Problem - we are getting Throttling: Rate exceeded exception for invalidation request. we observe that first 6-7 request executed successfully then we receive this error.
According to AWS documentation - we can have invalidation requests for up to 3,000 files per distribution in progress at one time. but we are receiving error after 6-7 request.
**Exception**
*Throttling: Rate exceeded
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'Throttling',
time: 2021-12-07T08:13:27.058Z,
requestId: '*****',
statusCode: 400,
retryable: true
}*
SDK version number v2.981.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 5
- Comments: 17 (4 by maintainers)
Hello @LYTzeng and @kellertk , I have written a small reverse engineering test script with sdk (typescript) where I invalidate a single path in sequence, wait X ms and repeat this a maximum of 30 times. In this test, I observe “Rate Exceeded” errors every time when X is less than 1600 ms. It would be good if such a major restriction were (officially) documented somewhere.
Hi @vudh1, Do we have any update on this issue?
Briefly
I tend to think it’s rather an CloudFront API issue than JS SDK issue. Or that there’s some coincidence of those requests being sent interlaced with some other and that’s why it hits throttling (but that’s a pure speculation)
@dipakjain94 have you tried passing
maxRetriesoption? Does it help?In detail
If there were too many paths to invalidate it should respond with
BatchTooLargeI guess. So that’s not it. If number of invalidations being processed was too high that’d beTooManyInvalidationsInProgressIt seems as if it was about endpoint being called too frequently. @dipakjain94 is the code you show called in some kind of loop maybe?
The error is
retryablethen SDK problem might be that it hasn’t been retried instead it was finalized. Might because we don’t know if it was retried or not. Or how many times. We may assume that it has been retried and that it was the default number of times.In the code sample we see that service has been instantiated with no options, so
maxRetriesshould be “calculated by the specific service object that the request is being made to”:https://github.com/aws/aws-sdk-js/blob/b0fd3156c162454a0bb644cca68349904cd2e52e/lib/config.js#L31-L34
So since it’s not set on
cloudfrontservice it’s 3:https://github.com/aws/aws-sdk-js/blob/b0fd3156c162454a0bb644cca68349904cd2e52e/lib/service.js#L169-L172
From what I see in the issue I can’t tell if there were retries or not in reported case and I haven’t searched for test confirming it works but if it didn’t then I guess it would have already been noticed.
@AllanZhengYP or @ajredniwja,
This issue still persists, could you please have a look?
Found this while searching for an issue at work… best I can put together from the AWS case we have open, there are actually “burst limits” (that’s actually the search term that led me here!). We were sending in no more than 70 Invalidations in an hour, but 10 (or so) of them were being requested in a single second. This seems to have triggered a “Rate exceeded” error.
Best I can tell, the solutions are: