serverless: More than one s3 event failing deployment for existing s3 bucket.

This is a Bug Report

Description

I have 30 different suffixes for which i want to trigger event associated with S3 and lambda. So i need to define all of them in serverless.yml file. It is working fine for one event but not working when i put two or more events. I think it is not working for multiple events.

What went wrong? Throwing following error:

Serverless Error ---------------------------------------

  An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. The statement id (GICreateThumbnails-gi3-staging-original) provided already exists. Please provide a new statement id, or remove the existing statement. See details in CloudWatch Log: 2019/07/27/[$LATEST]337e42c16b0643bcadd452556a6dc004.

What did you expect should have happened? I am expecting to see all events should be visible on my lambda console in AWS which i have defined in serverless.yml
What was the config you used?

service: User
frameworkVersion: ">=1.48.4"
tenant: ginvoicing
app: ginvoicing


custom:
  bucket: gi3-original

provider:
  name: aws
  runtime: nodejs8.10
  memorySize: 192
  timeout: 10
    iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:*
      Resource: "*"
  logRetentionInDays: 5
  stage: dev
  region: ap-south-1
functions:
  GICreateThumbnails:
    handler: GICreateThumbnails.handler
    name: GICreateThumbnails # optional, Deployed Lambda name
    description: To create thumbnail of the uploaded picture on s3. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket}
          event: s3:ObjectCreated:*
          rules:
            - prefix: images/items/
            - suffix: .png
          existing: true   
      - s3:
          bucket: ${self:custom.bucket}
          event: s3:ObjectCreated:*
          rules:
            - prefix: images/user/
            - suffix: .png
          existing: true

What stacktrace or error message from your provider did you see?

Stack Trace --------------------------------------------
 
ServerlessError: An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. The statement id (GICreateThumbnails-gi3-staging-original) provided already exists. Please provide a new statement id, or remove the existing statement. See details in CloudWatch Log: 2019/07/27/[$LATEST]337e42c16b0643bcadd452556a6dc004.
    at provider.request.then.data (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:121:33)
From previous event:
    at AwsDeploy.monitorStack (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/monitorStack.js:27:12)
    at provider.request.then.cfData (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:103:28)
From previous event:
    at AwsDeploy.update (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:103:8)
From previous event:
    at AwsDeploy.BbPromise.bind.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:117:35)
From previous event:
    at AwsDeploy.updateStack (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/lib/updateStack.js:113:33)
From previous event:
    at AwsDeploy.BbPromise.bind.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:127:39)
From previous event:
    at Object.aws:deploy:deploy:updateStack [as hook] (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:123:30)
    at BbPromise.reduce (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:55)
From previous event:
    at PluginManager.invoke (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:22)
    at PluginManager.spawn (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:484:17)
    at AwsDeploy.BbPromise.bind.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:93:48)
From previous event:
    at Object.deploy:deploy [as hook] (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/plugins/aws/deploy/index.js:89:30)
    at BbPromise.reduce (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:55)
From previous event:
    at PluginManager.invoke (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:464:22)
    at PluginManager.run (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/classes/PluginManager.js:496:17)
    at variables.populateService.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/Serverless.js:116:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/lib/Serverless.js:103:74)
    at serverless.init.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/bin/serverless.js:52:28)
    at /Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:111:16
    at /Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:45:10
    at FSReqWrap.oncomplete (fs.js:135:15)
From previous event:
    at initializeErrorReporter.then (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/bin/serverless.js:52:6)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Object.<anonymous> (/Users/tarunjangra/.nvm/versions/node/v8.11.3/lib/node_modules/serverless/bin/serverless.js:38:39)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:191:16)
    at bootstrap_node.js:612:3

Similar or dependent issues:

#6369

Additional Data

Serverless Framework Version you’re using: 1.48.4
Operating System: darwin
Enterprise Plugin Versione: 1.3.3
Platform SDK Version: 2.1.0

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 41 (12 by maintainers)

Most upvoted comments

@tarunjangra @michelem09 @HumbleBeck thanks for the in-depth error reporting.

I was finally able to reproduce the issue. It was caused by a race condition. I updated the PR which now includes the fix and the error seems to be resolved (I also added an integration test to cover this case). I know it might be annoying, but can one of you maybe test this PR again. TBH I’m not sure if there’s a way to test this deterministically since we already deployed some many different versions in the past which failed one way or another and the stack might be in a weird state. Anyway, I would be really grateful if one can jump in and give it a shot. 🙏

Thanks again for taking the time to take a deep dive into this issue. It’s certainly not that straightforward but I believe we’re onto something here! 👍 💯

pmuens on Aug 2, 2019

@pmuens I tested it thoroughly and got new issue. I am not sure if i should create new issue or should open the same one. Just to make you sure, I have installed master branch and it already has that fixed branch merged.

serverless deploy --verbose

Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service GIImageProcessing.zip file to S3 (935.89 KB)...
Serverless: Uploading custom CloudFormation resources...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - GIImageProcessing-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - GICreateThumbnailsCustomS31
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - GICreateThumbnailsCustomS31
CloudFormation - CREATE_FAILED - Custom::S3 - GICreateThumbnailsCustomS31
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - GIImageProcessing-dev
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - GIImageProcessing-dev
CloudFormation - DELETE_SKIPPED - AWS::Lambda::Version - GIDeleteThumbnailsLambdaVersionEuE9vug8attALyPDc62yVYPXcnJKrbBc4a5kqGVlk
CloudFormation - DELETE_SKIPPED - AWS::Lambda::Version - GICreateThumbnailsLambdaVersionhjRhQtuMscRr1F6MS9FvGWgqaRGyyzWrerjXScGhcGY
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - GIDeleteThumbnailsLambdaFunction
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - GIDeleteThumbnailsLogGroup
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - GICreateThumbnailsLambdaFunction
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - GICreateThumbnailsLogGroup
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - DELETE_COMPLETE - AWS::CloudFormation::CustomResource - GICreateThumbnailsCustomS31
CloudFormation - UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - GIImageProcessing-dev

serverless.yml

service: GIImageProcessing
frameworkVersion: ">=1.48.4"

custom:
  enterprise:
    collectLambdaLogs: false
  bucket:
    dev: gi-mu-staging
    prod: gi-mu-production


provider:
  name: aws
  runtime: nodejs8.10
  memorySize: 192
  timeout: 10
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:*
      Resource: "*"
  logRetentionInDays: 1
  stage: ${opt:stage,'dev'}
  region: ap-south-1
functions:
  GICreateThumbnails:
    handler: src/GICreateThumbnails.handler
    name: GICreateThumbnails # optional, Deployed Lambda name
    description: To create thumbnail of the uploaded picture on s3. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .png
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpeg
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectCreated:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpg
  GIDeleteThumbnails:
    handler: src/GIDeleteThumbnails.handler
    name: GIDeleteThumbnails # optional, Deployed Lambda name
    description: To Delete thumbnail of the image on s3 bucket. # optional, Description to publish to AWS
    events:
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .png
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpeg
      - s3:
          bucket: ${self:custom.bucket.${self:provider.stage}}
          event: s3:ObjectRemoved:*
          existing: true
          rules:
            - prefix: uploads/
            - suffix: .jpg

Error I am getting now.

Serverless Error ---------------------------------------
 
  An error occurred: GICreateThumbnailsCustomS31 - Failed to create resource. Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type. See details in CloudWatch Log: 2019/08/08/[$LATEST]1fe4381b53114851a8280926fe6ade69.
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              8.11.2
     Framework Version:         1.49.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0

tarunjangra on Aug 8, 2019

@tarunjangra it was deployed on another branch npm i -g serverless/serverless#existing-s3-multi

michelem09 on Jul 31, 2019

Thanks for the update @michelem09 👍

Yes, I too see this error when I upgrade from an old deployment. Right now it’s just working if you deploy from scratch. I try to mitigate this shortcoming so that you can upgrade without running into this issue.

pmuens on Jul 29, 2019