Hey everyone!

We’ve just prioritized this issue since it’s a problem many developers face when they work with complex services in a serverless context.

We’ve created a new issue (#3411) where we’ll gather different approaches how we can resolve this issue.

Would be really nice all of you could chime in on this and provide some feedback regarding the best / your favorite solution!

This is a major issue that is made worse by version 1.3.0. Since 1.3.0 adds versioned lambda functions it effectively creates once extra resource for every Lambda function [1] which for a large project can be a lot of resources. In mine it put me from 20 bellow the limit to 10 above just by upgrading.

Yes, there are ways to restructure the project but this is really something Serverless should take care of because it grinds projects to a screeching halt.

[1] https://gist.github.com/andrewcurioso/58bba11ac1175c26f508888ca466d0ea

@davestone Not sure why I got singled out for a solution since I’m not part of the Serverless team and my post was essential a “me too” with a note about why v.1.3.0 makes it worse but since you asked:

I think the most straight forward solution is to supported Nested CloudFormation stacks and allow configuration options on resources and functions in serverless.yml to specify which stack the resource should be included in.

That way small projects remain unchanged and large projects can leverage CloudFormation Nested Stacks in a way that makes sense for them. I can even think of uses for Nested Stacks besides just circumventing the 200 resource limit. For example, making deploys faster.

Edit: This, of course, does not directly solve the issue. But at least gives a path to being able to solve it themselves. Another possibility is to automatically breakup the stack in some way. For example, since most of the resources created are tied to the lambda function you could break those out into groups. There would still be a limit but it would be more manageable.

Edit/Note 2: I hit the 200 resource limit with just 34 Lambda functions because I have DynamoDB tables, SNS streams, etc.

I’d propose the management of lambda’s is taken out of CloudFormation. When we 1st developed JAWS I had a CF file for labmdas and a CF file for everything else for this reason (among others). We eventually decided to deploy lambdas via API for speed of deployment, limits of CF and complexity of breaking up and managing multiple CF files.

I struggle with depending on CloudFormation for any “newer” services from AWS (like lambda, APIG, CloudFormation) because the CF team really struggles with keeping up on new features (rightly so, there are feats being released all the time).

@Si1kIfY: A get-to-production solution is to break the project into multiple sensibly separated services. Note that if you need a single ApiGateway, you will need to keep all functions hanging off of it in the same service. The generated template may be reducible if you use custom IAM role(s), defining those in a separate service along with LogGroup declarations for each of your functions. Additionally, you can move any of the non-function resources to the separate (or another) service. Deployment would consist of an in-order deployment as it makes sense on the basis of resource dependencies.

From a how do I look into and gain this knowledge myself/look under the covers standpoint, use the --noDeploy flag and look into the ~/.serverless/cloudformation-template-*-stack.json files. The role and log groups are generated unless you supply pre-created roles for all functions (specifically at mergeIamTemplates.js, line 27).

@felipefdl Yes, it does look like you have hit the same error as us. I don’t think that the note you posted is directly related to this. The template resource error is actually related to a CloudFormation limitation for deployment. It actually has nothing to do with APIG specifically.

The only work arounds for Serverless v1.0+ is to break up your project into smaller services, or place it in some kind of a wrapper so use fewer endpoints. There is no out of the box solution. I don’t believe there is an ETA for an official fix on this since the core team has not labeled it as a priority (or even recognized it as a significant problem).

Should the framework do all of this?

@nicka I strongly believe that it should. Serverless is attempting to ease the deployment of services, and it does it very well as long as your services are tiny. As soon as you have a real project these hard limits make it unusable. It should be just as easy to deploy 10 endpoints as it is to deploy 100.

I’ve started work on a plugin to address this issue (among others), as we are also unable to use v1 as it is architected. In short using CloudFormation to manage API Gateway and Lambda plain does not work for us (README.md in my plugin goes into specifics on why). IMO current v1 is good for development, but not a real world production environment. We are stuck on v0.5 for our existing workloads.

We prefer to let swagger be the interface to manage APIG and direct API integration to create and update Lambdas, while keeping permissions (IAM) and other not frequently changing resources in CloudFormation (or maybe Terraform).

Does anyone have similar sentiments?

@jordanmack is right! Depending on your setup, another solution could be grouping functions and using something like serverless-http.

functions:
  foo:
    handler: handler.foo
    events:
      - http: # Catch all proxy useful for express/koa apps
          path: '{proxy+}'
          method: any

@asantibanez @mwawrusch We’re currently splitting our stacks(services) with Fn::ImportValue’s as they are the most flexible(for us). Please check (https://www.youtube.com/watch?v=TDalsML3QqY) from 13:27.

Should the framework do all of this? It would definitely be possible for the framework to do all the heavy lifting for you, but IMHO this would become very complex fast. As a developer I think you’d rather have control over this your self (think about order of stack deployments etc.).

@jordanmack I’ve begun work on a new serverless plugin https://github.com/doapp-ryanp/serverless-plugin-swag . I haven’t spent much time on it yet as I wanted to see what was going to be announced @ re:invent. Honestly I’m not sure when I will get time to work on it in the next few weeks - prob wont be til beginning of next year. But I can assure you I have the same issues (and more) as you so I will have to address them at somepoint.

@johncmckim yea my question was more in regard to decoupling serverless v1 from cloudformation for create/update of APIGateway . I have a substatial list of other reasons on why, but since it was large I didn’t want to hijack this thread but I think for clarity I’ve changed my mind, and will put it inline:

• CloudFormation has item limits (this issue)
• It is not a security best practice to allow deletion of assets from an AWS API Key (can’t back with MFA)
• CloudFormation is very powerful. We like to visually validate (via diff and CF web UI diff function) exactly what is going to change, not hide it in the framework internals.
• Serverless deletes and re-creates APIG distro on every update
• CloudFormation is not good for frequently changing assets because it sometimes gets into a update/rollback failed state. You must delete CF and re-create. Not fun hitting this in production
• CloudFormation does not stay on top of new AWS features