aws-cdk: (lambda): Get Function code + layer exceeds maximum size error when it does not
The Lambda layer is being applied before code changes when updating. Then the max size limit of the lambda is reached when deploying. This is either an CDK specific issue or AWS Cloud Formation, I could not find any evidence of this on the internet and consider it to be a fringe case.
Reproduction Steps
I have a deployed lambda function that is 230MB. Then after making changes to the lambda function it is reduced to ~50MB. A lambda layer is applied to the this function ~130MB. The combined size of the lambda + layer is then ~180MB which is less than the 250MB limit. When I try to deploy this I get the following cloudformation error:
...
20/101 | 1:27:01 PM | UPDATE_FAILED | AWS::Lambda::Function | <LambdaFunctionName> (XXX) Function code combined with layers exceeds the maximum allowed size of 262144000 bytes. The actual size is 361955628 bytes. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: XXX)
...
Which looks like the Layer is applied before the code update (as the new code + layer is less than 250MB), and then fails the size constraint. The actual size it reports is the old code + layer: 230MB + 130MB = 360MB which lead me to this fringe case conclusion.
It deploys when you completely destroy the stack and then use the layer + new code. Which is the same as destroying the lambda function or manually updating the lambda with the new code first and then running the update using cloud formation/CDK.
Environment
- CLI Version : 1.26.0
- Framework Version: 1.32.0
- Node.js Version: v10.16.3
- OS : Windows & Linux
- Language (Version): TypeScript
This is š Bug Report
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 12
- Comments: 28 (12 by maintainers)
We are also still experiencing this issue, and itās causing quite a bit of pain with our deployments.
I faced the same issue today and wanted to share my work around.
I renamed the function that is causing the issue e.g.
yourfunctionV2. This creates a new function and removes the old one. This way, you donāt have to manually deploy again to add layers.Just confirming that Iām still experiencing this issue in 2021. Thought I was going slightly crazy until I found this.
In my experience, I had a few lambda functions that were each very fat, each having a copy of some large shared libraries. Naturally I wanted to refactor to extract the shared libraries into a layer, however after doing so, I ran into this problem where even though my functions were now tiny and the layer was fat, my
cdk deploywould tell me that:Even though what I was seeing in the build was more like:
And this was my whole deployment, so I couldnāt understand how I was breaching the 250 MB limit.
To resolve it, I had to
cdk destroyandcdk deployagain which is disappointing.Steps to reproduce from my experience are therefore something like:
cdk deploycdk deployWe ran into this over the last few days. Our bundle is ~8MB zipped and the layer is ~15MB zipped, but we kept getting this error:
Using @jeffskiās suggested workaround seemed to do the trick:
I went through the serverless template and renamed every defined Lambda and the deploy worked. A subsequent deploy reverting the name changes then also worked.
Also, I was tagged here 4 years ago when I was on the Lambda Team (moved on to APIGW recently, so close by š ) and have never checked this until I got a notification today š¦
For clarity:
The fundamental problem, is what @roryzhg pointed out: Itās alllll in the APIs. Lambda CloudFormation resource only calls Lambda APIs, so if one canāt create a combination of API calls to sort out the limitation, then itās unsolvable in CloudFormation.
The API itself is flawed:
There are multiple ideal solutions:
UpdateFunctionCodeand deprecate the usage inUpdateFunctionConfiguration. This one would be a bit cleaner, as then you have all the necessary resources in one call, so itās atomic counting, vs the split brain knowledge today.coderelated bits intoUpdateFunctionConfiguration, therein deprecating the usage ofUpdateFunctionCode, and makingUpdateFunctionConfigurationa single atomic call with all data references.I would suggest reaching out to Lambda support cases, and linking my message, so hopefully they take those suggestions to fix this egregious miscalculation in the future.
Have run in to this a couple of times now and wanted to share the workaround that worked for us.
We are deploying using the Serverless Framework and essentially what we do is rename the Lambda in the config file. This creates a brand new Lambda, instead of trying to modify the existing Lambda. We are running the Lambda in a Step Function so we update that to use the new Lambda name.
This all seems to work although with minor disruption while the changeover happens, I think due to the way things align in Step Functions. It is preferable to removing and re-adding layers or doing a remove/redeploy as our deployments take several minutes and would result in considerable down time.
Anyway, this might be an option for anyone in this situation and might work with other triggers.
New ref for internal ticket: P80248897. Unfortunately, we canāt do much about this issue from the CDK perspective.
The internal ticket raised by Niranjan still exists and hasnāt been engaged with yet. Iāve reached out to the team about this to hopefully get some engagement
Bumping this! I encountered the same issue. Itās very confusing. I had to do 2 deployments to get around this issue