serverless: v0.5 - Error when rate limited by AWS API

I’m seeing an issue with deploying all endpoints and hitting the API Gateway rate limit. I believe Serverless is set up to sleep for 5 seconds after getting a 429 error status and then retrying the request, but I get this error:

Wed, 23 Mar 2016 20:49:58 GMT serverless:lib.node_modules.serverless.lib.actions.EndpointDeployApiGateway 'Too many requests' received, sleeping 5 seconds
/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:49
        fn = function () { throw arg; };
                           ^

ServerlessError: No integration defined for method
    at new ServerlessError (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Error.js:17:11)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:461:25
From previous event:
    at EndpointDeployApiGateway._createDeployment (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:424:14)
    at EndpointDeployApiGateway.<anonymous> (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:213:24)
From previous event:
    at EndpointDeployApiGateway._processRegionDeployment (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:207:10)
    at EndpointDeployApiGateway.<anonymous> (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:94:24)
From previous event:
    at EndpointDeployApiGateway._processDeployment (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:91:10)
From previous event:
    at EndpointDeployApiGateway.endpointDeployApiGateway (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:55:10)
From previous event:
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:206:25
    at Array.reduce (native)
    at Serverless._execute (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:205:26)
    at Object.actions.(anonymous function) [as endpointDeployApiGateway] (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:424:20)
    at EndpointDeploy._processDeployment (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeploy.js:212:24)
From previous event:
    at EndpointDeploy.endpointDeploy (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeploy.js:101:12)
From previous event:
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:181:31
    at Array.reduce (native)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:180:32
From previous event:
    at Serverless._execute (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:178:12)
    at Serverless.actions.(anonymous function) (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:424:20)
    at Serverless.command (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:393:38)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/bin/serverless:19:16
    at processImmediate [as _immediateCallback] (timers.js:383:17)
From previous event:
    at Object.<anonymous> (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/bin/serverless:18:4)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
    at Function.Module.runMain (module.js:430:10)
    at startup (node.js:141:18)
    at node.js:980:3

There was a bug related to dealing with rate limit errors in version 0.4 that was fixed in #688, but now it looks like there’s a new problem in version 0.5.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 19 (10 by maintainers)

Most upvoted comments

I can confirm the latest version in master fixes the issue I was seeing.

I did a full deployment of all endpoints in my project, and on every 429 response Serverless slept for 5 seconds before retrying successfully. It’s working as it should.

marekmenhart-boa on Apr 22, 2016

UPDATE: Seems to be fixed by this PR https://github.com/serverless/serverless/pull/1025

HyperBrain on Apr 22, 2016

Thanks for your help @pmuens and @HyperBrain, I’ve opened 9cookies/serverless-aws-documentation#34 to follow up on this.

Best!

marcelog on Apr 6, 2017

Yes, that can be caused by the plugin. Especially models generate lots of additional resources in the generated CF template. If they are not made dependent by the plugin to serialize the creation by CloudFormation, it can happen that CF internally receives a 429 (Too many requests) error, from which it cannot recover.

HyperBrain on Apr 5, 2017

Hi @pmuens,

So this is not currently happening, but it did happen in 3 different times right at the end of the deploy (i.e: everything was being deployed correctly, and just before returning to the shell, I got the error after the “Stack Outputs” part).

I’m trying to reproduce it so I can post the logs. I’m using the serverless-aws-documentation plugin and I do have a lot of models, I’m not sure if plugins can trigger this error.

I’ll keep you guys posted if it happens again. Thank you 😃

marcelog on Apr 5, 2017

@eahefnawy Thanks for your response. I’m able to deploy the same project without any issues as long as I don’t hit the rate limit.

I noticed I get a different stack trace depending on whether I include the --all flag. My original error stack trace was generated after running the command

serverless endpoint deploy --all -s <myStage> --debug

If I don’t include the --all flag and run this instead

serverless endpoint deploy -s <myStage> --debug

I get this output:

Serverless:   POST - /dspCloud/govern/1_0/{companyId}/customAttributes/values: Too Many Requests  
Mon, 28 Mar 2016 16:25:57 GMT serverless:lib.node_modules.serverless.lib.actions.EndpointDeploy TooManyRequestsException: Too Many Requests
    at Object.extractError (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/protocol/json.js:43:27)
    at Request.extractError (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/protocol/rest_json.js:37:8)
    at Request.callListeners (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:596:14)
    at Request.transition (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:21:10)
    at AcceptorStateMachine.runTo (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:37:9)
    at Request.<anonymous> (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:598:12)
    at Request.callListeners (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:596:14)
    at Request.transition (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:21:10)
    at AcceptorStateMachine.runTo (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/aws-sdk/lib/state_machine.js:26:10
From previous event:
    at ServerlessProviderAws.request (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/ProviderAws.js:65:14)
    at ServerlessProviderAws.getApiByName (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/ProviderAws.js:302:19)
    at Builder._getRestApi (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointBuildApiGateway.js:193:18)
From previous event:
    at Builder.build (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointBuildApiGateway.js:71:10)
    at EndpointBuildApiGateway.endpointBuildApiGateway (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointBuildApiGateway.js:53:22)
    at processImmediate [as _immediateCallback] (timers.js:383:17)
From previous event:
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:206:25
    at Array.reduce (native)
    at Serverless._execute (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:205:26)
    at Object.actions.(anonymous function) [as endpointBuildApiGateway] (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/Serverless.js:424:20)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:172:36
From previous event:
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:146:35
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/async/lib/async.js:181:20
    at iterate (/scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/async/lib/async.js:262:13)
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/async/lib/async.js:274:29
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/node_modules/async/lib/async.js:44:16
    at /scratch/jenkins/tools/jenkins.plugins.nodejs.tools.NodeJSInstallation/Node_Default/lib/node_modules/serverless/lib/actions/EndpointDeployApiGateway.js:199:26
    at processImmediate [as _immediateCallback] (timers.js:383:17)

The same error is given for every subsequent endpoint in my project. It doesn’t look like there’s any attempt to sleep and retry, it simply logs the failure and proceeds to the next endpoint.

marekmenhart-boa on Mar 28, 2016