serverless: Provisioned Concurrency Does Not Work As Described

Bug Report

Provisioned Concurrency was recently added in Serverless v1.59 and described in detail in this blog post: https://serverless.com/blog/aws-lambda-provisioned-concurrency/

The blog post makes the claim that enabling provisioned concurrency is as easy as:

functions:
  hello:
    handler: handler.hello
      events:
        - http:
            path: /hello
            method: get
    provisionedConcurrency: 5

This is not true.

What this configuration will do is assign provisioned concurrency to the version number that was deployed, but API Gateway will continue invoking the Lambda with the $LATEST alias (no version). What effectively ends up happening is every deploy keeps assigning provisioned concurrency to each new version and never actually utilizes it. If a developer does not use any version pruning they will quickly rack up an expensive bill paying for new provisioned capacity on each version, and potentially worse, eventually exhaust their reserved (or unreserved) concurrency on their AWS account.

I think this is a critical issue and needs to, at the very least, be addressed in the blog post. Without some form of version cleanup or aliasing I don’t think there’s much Serverless can do to make this work in a simple configuration like that.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 33
  • Comments: 38 (16 by maintainers)

Commits related to this issue

Most upvoted comments

I’ll reopen until we have all issues mentioned here solved.

+7
medikoo on Dec 12, 2019

@vavarodrigues it’s still not really fixed, but it should be shortly with v1.60.2, stay tuned.

+6
medikoo on Dec 20, 2019

@AndrewBarba great thanks for reporting that issue

We will shortly (hopefully tomorrow) publish an update that fix the behavior in a framework (to make blog post accurate).

+6
medikoo on Dec 5, 2019

@medikoo My obervations are following (severless version: 1.59.3)

  1. It is not working on existing stack as its creating new concurrency for each deployment.
  2. On a new stack: a. It works for the first time b. If I modify the provisioned concurrency (from 5 to 6) then cloud formation update failing (UPDATE_FAILED: Internal Failure)

These issues making it really hard to use the provisioned concurrency in serverless

+3
shaheer-k on Dec 12, 2019
  1. It is not working on existing stack as its creating new concurrency for each deployment.
  • v1.59.3 won’t delete provisioned concurrency configuration for versions published with v1.59.0-2
  • v1.59.3 on first deploy will introduce a new special and only version on which provisioned configuration is configured (it’s definitely not the case that with each following deploy you’d have new versions created)

b. If I modify the provisioned concurrency (form 5 to 6) then cloud formation update failing (UPDATE_FAILED: Internal Failure)

This is the issue on AWS side (note Internal Failure comes from CloudFormation stack update, you can also see it in AWS console). We’ve communicated that issue to AWS, still we have no response on that so far.

I’m clarifying with AWS, the best way to handle that, if going through CloudFormation will be communicated as not stable (for prolonged period), we’ll probably reconfigure our support to go through custom resources.

+2
medikoo on Dec 10, 2019

I’ve since updated, so the problem may be fixed.

We had many fixes related to provisioned concurrency support, since that version. Always ensure to use latest version of a Framework

+1
medikoo on Dec 4, 2020

So I didn’t add --alias provisioned while deploying , do you mean this affects ?

provisioned alias is configured by Framework automatically, note that provisioned concurrency can only be set on alias (and it won’t work for $LATEST)

Therefore to inspect provisioned concurrency in AWS console, you need to visit a provisioned alias for your lambda

+1
medikoo on Apr 28, 2020

@RogerVFbr i went back to using https://github.com/AntonBazhal/serverless-plugin-lambda-warmup until the provisioned currency issues have been resolved

+1
jplock on Dec 11, 2019

Problem seems to persist. Any ideas if a fix is underway? This is very important for our project. Provisioned concurrency still seems to be pointing to a function version instead of $LATEST. “versionFunctions” has been set to “false” and alternative versions have been cleaned via script. However running “sls deploy” with some “provisionedConcurrency” value attached to the function declaration creates an alternative function version and points the provisioned concurrency to it. Redeploying the project does not seem to affect the situation. Is it possible that the version stated is equivalent to $LATEST, therefore it would be working? Can anybody please show a way to test if the invocation of latest is making use of the provisioned concurrency with this configuration?

Anotação 2019-12-11 151441

Anotação 2019-12-11 150936

Anotação 2019-12-11 150911

+1
RogerVFbr on Dec 11, 2019

Definitely some issues. Subsequent deploys leave the provisioned concurrency configuration attached to the special version that is not $LATEST as @medikoo says above.

Also to get this to work with an existing lambda, I had to delete all the versions then set versionFunctions to false. You can use this handy script if you’re having issues with that: https://gist.github.com/tobywf/6eb494f4b46cef367540074512161334

+1
dkobia on Dec 10, 2019
Read more comments on GitHub
← serverless: Profile [profilename] does not exist. Using Cross account roles. credentials_source=EcsContainer
serverless-python-requirements: 4.2 re-orders requirements.txt before installing, breaking extra pypi index →