serverless: Rate exceeded

Ok. Then this is a severe bug in the variable resolution part of Serverless. Upon invocation of the lifecycles it MUST be guaranteed that the whole serverless.yml file is resolved with each of its variables that it contains as well as any references that there may be.

@horike37 @pmuens @RafalWilinski We should open a separate issue for that with high priority. https://github.com/serverless/serverless/issues/3821#issuecomment-336169219 shows clearly that there is a bug. We have to make sure that the variable resolution finishes (and SLS waits for that) and only then continues to start the command lifecycles.

@pmuens I think it is time to do the serialized requests now. @gozup 's issue clearly shows that this will be the only way to eliminate the problem forever. Every other approach that still allows to break the limit of parallel AWS REST API accesses is not really a solution 😃 but merely a workaround that will make the problem more obscure with each PR.

I’m not sure how to implement that at this time.

I think the right place would be the API request method itself, as it is used centrally from every location of the framework. It could be handled by using a promised queue which just queues submitted method call promises (I think a proper module was BbQueue)

UPDATE: It is bluebird-queue (https://www.npmjs.com/package/bluebird-queue). This even allows to set a concurrency limit, so that we can go straight to the limit without exceeding it.

@ozbillwang Yes. With this fix, referencing variables with ${cf:...} would work again. The difference is, that the value of the referenced output is only retrieved during build time and thus does not go under CloudFormation control and it does not enforce a CF dependency to the stack from which the output is retrieved.

@ozbillwang You have to use Fn::Join instead of Fn::Sub as that will clash with Serverless’ variable notation. Try to use:

Outputs:
    DynamoDbTable:
      Value:
        Ref: DynamoDbTable
      Export:
        Name:
          Fn::Join:
            - ''
            -
              - Ref: AWS::StackName
              - '-DynamoDbTable'

Hi,

I also receive this error however with only one ${cf:} variable (for a user pool which I assign to a custom variable that is then referenced as the authorizer for each lambda function). I can’t use Fn::ImportValue for this presumably because sls needs the value for the authorizer at the time of packaging.

It seems that each time a custom variable is referenced sls retrieves the value from aws. Also if you have multiple variables in an object that reference ${cf:} vars, then any time you reference that object it looks like all of the ${cf:} variables inside that object are retrieved. This obviously can quickly lead to a lot of requests and the corresponding ‘Rate exceeded’ error (I can get it with just one ${cf:} variable that is referenced a lot). My testing suggests I can get between 21-23 aws requests in quick succession before getting the rate exceeded error.

My quick solution was to change the code in Variables.getValueFromCf to return a cached promise if the variable had already been requested, i.e.:

getValueFromCf(variableString) {
  ...
      if (
      this.cfVars &&
      this.cfVars[stackName] &&
      this.cfVars[stackName][outputLogicalId]
    ) {
      return this.cfVars[stackName][outputLogicalId];
    }
    let promise = this.serverless.getProvider("aws").request(...);
    if (!this.cfVars) {
      this.cfVars = {};
    }
    if (!this.cfVars[stackName]) {
      this.cfVars[stackName] = {};
    }
    this.cfVars[stackName][outputLogicalId] = promise;
    return promise;
}

Much appreciated, @pmuens 😍

Great! That should do the trick for no. Furthermore that should give you more security cross-referencing your stacks.

We’ll investigate in this one and try to fix it so you can switch back to ${cf:} later on (if you’d like to).

Yep, I unfortunately could not deploy at all so I’ve hardcoded some values for the moment 🤢

🤔 what about using Outputs an Fn::ImportValue for the time being?

You’re welcome 😄 . Yes, according to the stack it’s there.

@brunobelotti “Rate exceeded” must originate from Serverless’ deployment phase (uploading the artifacts and calling the CF deployment) as serverless-webpack only does its work during the package phase (where no external interaction is involved).

So any plugin that hooks into the deployment phase is a candidate as well as Serverless itself. The error is catched by Serverless’ exception handler, so I think it is an AWS call. Can you try to execute SLS_DEBUG='*' serverless deploy ... to get a proper stacktrace for the “Rate exceeded” error. Then we should be able to exactly trace back where the error happens?

FYI (different topic): I saw that you use yarn to deploy (and probably setup the node_modules folder). Serverless as well as any released serverless-webpack versions use npm for packaging and might mess up the dependencies installed with yarn - and they do not respect any yarn lock file. Serverless-webpack (master branch) contains full yarn support now to overcome these issues, so you should prepare to update with the next release there or try the master branch.

I removed .cache folder from node_modules and it seems to work. The error appears sometimes… In a few days I’ll back to comment

The complementing PR is finished now. I’d appreciate if people in this thread would test out the cache-AWS-requests branch ("serverless": "github:serverless/serverless#cache-AWS-requests") and report here if all their rate exceeded issues in the variable system are resolved now.

Hi Bruno,

I saw you opened an issue on the plugin’s repo. I’ll dig into it by the end of the day.

However, you can be sure that it works as I’m using it in production on a large project that fetches more than 30 secret parameters.

Seems like the problem is more coming from the installation than from the plugin itself.

Cheers, Emmanuel

Le 27 oct. 2017 à 06:41, Bruno Watt notifications@github.com a écrit :

@HyperBrain where can we find the separate issue you mention for fixing this bug? This is a huge nightmare and the mentioned plugin does not work in any stack i try it in.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Hi @pmuens, @HyperBrain, @horike37, @RafalWilinski,

FYI, I just did a pull request of my plugin to handle SSM parameters in the meantime. MR: https://github.com/serverless/plugins/pulls Plugin repo: https://github.com/gozup/serverless-ssm-fetch

Hope it helps.

Cheers, Emmanuel

Good catch @gozup 👍

Yes, @HyperBrain AFAIK we have the same issue with the cf and the s3 Serverless Variables implementation (especially cf).

It would be nice if we could “queue and batch” the requests somehow. AFAIR we’ve discusses such a behavior in one issue (couldn’t find the correct reference right now) 🤔

Edit: Hahaha. I finally found the reference. Here you go: https://github.com/serverless/serverless/issues/3821#issuecomment-335494259 😂

Doh! Silly me, completely forgot about those, I’ll give that a go!!

Much appreciated, @pmuens 😍

+1 having the same issue with 3 ${cf:}-referenced variables