amplify-cli: Limit on the number of resources in a single stack operation exceeded (V2 Transformer Migration)
Before opening, please confirm:
- I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
- I have searched for duplicate or closed issues.
- I have read the guide for submitting bug reports.
- I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
- I have removed any sensitive information from my code snippets and submission.
How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v17.1.0
Amplify CLI Version
7.6.19
What operating system are you using?
Linux
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
No
Amplify Categories
Not applicable
Amplify Commands
push
Describe the bug
I am trying to migrate a large system from V1 to V2 Transformer. I have upgraded ElasticSearch 6.2 to OpenSearch 1.1. I have performed the “amplify migrate api” and stripped away all the custom resolvers we have.
The amplify push gets a long way through the deployment, but ultimately fails with the message “Limit on the number of resources in a single stack operation exceeded”.
None of my individual stacks are above 500 resources, so I figure this is something to do with concurrent resources. So I checked the current limits on my account, but don’t see any way to change that (have opened up a ticket with the CloudFormation Team):
$aws cloudformation describe-account-limits
ACCOUNTLIMITS StackLimit 2000
ACCOUNTLIMITS StackOutputsLimit 200
ACCOUNTLIMITS ConcurrentResourcesLimit 8500
Not sure how to proceed here.
Expected behavior
Push does not fail
Reproduction steps
See above
GraphQL schema(s)
# Put schemas below this line
Log output
# Put your logs below this line
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 32 (12 by maintainers)
Hi @undefobj / @josefaidt - appreciate your updates to status
Hi @edwardfoyle - just wondering if you have any feedback on this one. Its going to prevent anyone with a significant database migrating to the V2 platform (of course in my opinion). I know we are stopped in our tracks. So any viable workaround also appreciated.
Hi @SebSchwartz - thanks for the input. I’m actually doing exactly that with the connectionStack.json (see table in the above comment (https://github.com/aws-amplify/amplify-cli/issues/9762#issuecomment-1040758507) - splitting the 1,281 resources in connectionStack.json into 4 files with transform.conf.json - and as can be seen, that is working fine. That however is not this issue.
The issue is that the TOTAL number of resources across all stacks can’t exceed 2,500 resources. To deal with this, Amplify needs to “…split the stacks and use cross stack references…”
We have just hit this too.
We moved from V1 to V2 and have a large GraphQL deployment at the moment. ( 45 models ). It almost gets to the end of the deployment and fails on a stack with:
Limit on the number of resources in a single stack operation exceededWe are using the CDK output functionality so I was able to get a count and it seems that the deployment is attempting to deploy 74 CloudFormation templates with 2604 resources over all of them.
Ah, yes, I see it in
/build/stacksnow.FWIW, I tried again without the
Create*Resolverand instead added some assorted relational resolvers, and this appears to have worked: It brought us under the 500 cap, and it built successfully. So knock on wood, I may be unblocked.@akshbhu I resent my schema it essentially breaks/creates all kinds of v2 migration issues.
Hi @akshbhu - I have emailed our schema to you. Thanks
Hi @akshbhu
we also have a large schema and are facing the same issue.
I have shared our schema with you via the mentioned mail address. Thanks.
Hi @undefobj / @josefaidt - just checking back to see if there are any updates on this? This is obviously a killer when migrating any significant system. Appreciate any feedback you can provide.
Hi @edwardfoyle - in the table below you’ll see that I’ve split stacks such that they are all less than the 500 maximum. With the V2 transformer however the number of resources, particularly in the connectionStack* is significantly higher. This means I have a total number of resources across all stacks is now 3,528.
Amplify submits this to CloudFormation as a nested stack. I opened a ticket with AWS CloudFormation support and they came back with this feedback.
“…There also exists a limit on the total number of resources for nested stack which is 2500 per chain and if it goes beyond that, we may encounter the aforementioned error message. If this is the case, you can also split the stacks and use cross stack references [5] - It exports an output from one stack and import it in another and you can access resources in a different stack, as long as they are in the same account and AWS Region…”
It would appear from this that with the significant expansion of resources with the V2 transformer that we are now exceeding (by a significant amount) the 2500 resource limit per nested stack - even if the individual stacks are less than 500 (as is my case).
Can Amplify be configured to support / enable cross stack references to move past this limitation. With the significant increase in resources caused by V2 it would appear that any production user with a significant database is going to hit this limitation pretty quickly.
Because it stops a production upgrade I’d put this in as a bug rather than a feature request as it is currently classified.
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta name=ProgId content=Excel.Sheet> <meta name=Generator content="Microsoft Excel 15"> <link id=Main-File rel=Main-File href="file:///C:/Users/SCRAMP~1/AppData/Local/Temp/msohtmlclip1/01/clip.htm"> <link rel=File-List href="file:///C:/Users/SCRAMP~1/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml"> <style> </style> </head> <body link="#0563C1" vlink="#954F72">