serverless: Setting existing:true on S3 bucket causes deployment to fail

Bug Report

Setting existing:true on S3 bucket causes deployment to fail

Description

  1. What did you do? Added existing:true to S3 function. The deployment works correctly for a new bucket.

  2. What happened? The deployment fails with:

...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - pdfcountdebug-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - S3uploadedLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - S3uploadedLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - S3uploadedLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleCustomResourcesLambdaExecution
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - S3uploadedLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - S3uploadedLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - CustomDashresourceDashexistingDashs3LambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - S3uploadedLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - S3uploadedLambdaVersioneyfdcHxB4Kq0aaxjAYGHiQwjGYOm5C3Tr4KbmAP3pY
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - S3uploadedCustomS31
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - S3uploadedLambdaVersioneyfdcHxB4Kq0aaxjAYGHiQwjGYOm5C3Tr4KbmAP3pY
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - S3uploadedLambdaVersioneyfdcHxB4Kq0aaxjAYGHiQwjGYOm5C3Tr4KbmAP3pY
CloudFormation - CREATE_IN_PROGRESS - Custom::S3 - S3uploadedCustomS31
CloudFormation - CREATE_FAILED - Custom::S3 - S3uploadedCustomS31
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - pdfcountdebug-dev
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - pdfcountdebug-dev
CloudFormation - DELETE_IN_PROGRESS - AWS::CloudFormation::CustomResource - S3uploadedCustomS31
CloudFormation - DELETE_SKIPPED - AWS::Lambda::Version - S3uploadedLambdaVersioneyfdcHxB4Kq0aaxjAYGHiQwjGYOm5C3Tr4KbmAP3pY
CloudFormation - DELETE_FAILED - AWS::CloudFormation::CustomResource - S3uploadedCustomS31
...

and then rolls back everything else. The rollback for S3uploadedCustomS31 fails of course.

  1. What should’ve happened? Deployment should complete. The step that fails is the custom resource handler that attaches the necessary policies to the function handler and the existing bucket. It is not clear which step fails. I suspect it is attaching the permission to the lambda.

  2. What’s the content of your serverless.yml file? Reduced file for clarity

service: pdfcountdebug

provider:
  name: aws
  runtime: nodejs10.x
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'sa-east-1'}

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:GetObject"
        - "s3:DeleteObject"
      Resource: "*"

functions:
  s3uploaded:
    handler: src/uploaded.handler
    events:
      - s3: 
          bucket: hookohm.test.doutore-documentacao
          event: s3:ObjectCreated:*
          rules:
            - suffix: .pdf
          existing: true
  1. What’s the output you get when you use the SLS_DEBUG=* environment variable (e.g. SLS_DEBUG=* serverless deploy) sls does not fail, but the AWS deployment does.

Similar or dependent issues:

  • #6638 This gave a different error.

Because the whole stack rolls back due to the error, there are no AWS logs to track down the issue. Debugging CloudFormation custom resources is a bit of a dark art.

  1. Environment
  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              8.11.3
     Framework Version:         1.53.0
     Plugin Version:            3.1.0
     SDK Version:               2.1.1
     Components Core Version:   1.1.1
     Components CLI Version:    1.2.3

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 25
  • Comments: 41 (13 by maintainers)

Most upvoted comments

try to check: s3 -> bucket -> configuration -> events delete existing events it helps when you cannot deploy sls application

@jazwiecki - The error you’re seeing it unrelated to S3 bucket - if you don’t specify a role for Lambda functions to use, one will be created for you by Serverless Framework and used by your functions. You can avoid that by specifying an existing role. You can read more about it here: https://www.serverless.com/framework/docs/providers/aws/guide/iam#iam

@d3m0n1n it appears that role you list at iam.deploymentRole doesn’t have "lambda.amazonaws.com" listed as one of the trusted principal services

Hi All,

I have something like that error, my configuration serverless it’s the following:

service: NAMESERVICE

provider:
  name: aws
  runtime: dotnetcore3.1
  timeout: 10
  autoPublishAlias: live
  region: us-east-2 # AWS region

  deploymentBucket:
    name: BUCKETDEPLOY

  iam: 
    role: NAMEROLE
    deploymentRole: DEPLOYROLE

  vpc:
    securityGroupIds:
      - SECURITYGROUPID
    subnetIds:
      - SUBNETIDID

package:
    artifact: CACHEPACKAGE
  
functions:
  eventRequest:
    name: NAMELAMBDA
    handler: HANDLERAPI
    events:
      - s3:
          bucket: NAMES3
          event: s3:ObjectCreated:*
          rules:
            - prefix: waves/
            - suffix: .csv
          existing: true

But at the time of deploying it gave me the following error:

Serverless Error ----------------------------------------

An error occurred: CustomDashresourceDashexistingDashs3LambdaFunction - Resource handler returned message: “The role defined for the function cannot be assumed by Lambda. (Service: Lambda, Status Code: 400, Request ID: f079df33-4ddf-4543-bb49-517b2fd43fc7, Extended Request ID: null)” (RequestToken: bff75f0e-1f65-795c-3b03-84f7192b2913, HandlerErrorCode: InvalidRequest).

Please, i need your help

If anybody faces the issue bellow AND is using custom cfn role for deployment, please read solution here after.

Serverless Error ---------------------------------------

ServerlessError: An error occurred: S3uploadedCustomS31 - Failed to create resource. Access Denied See details in CloudWatch Log: 2019/10/01/[$LATEST]0d3cb59b1731410db457c06299c4df35.

Ensure your custom cfn role has the following actions allowed :

  • the following 4 actions related to lambda trigger configuration handled by the customResource lambda doing provisioning.

    • s3:PutBucketNotification
    • s3:GetBucketNotification
    • lambda:AddPermission
    • lambda:RemovePermission
  • the following 2 actions to ensure cfn can create a role for the lambda above and pass its allowed actions to the newly created role

    • iam:CreateRole
    • iam:PassRole

In my case I had to change

    events:
      - s3:
        existing: true

to

    events:
      - s3:
          existing: true

(notice the two extra spaces)

It’s weird how the error message is so cryptic though

We are facing same error with existing:true , seems like current workaround is only using manual trigger creation in AWS Lambda console

We ran into this issue. Our deployment role does not have iam:CreateRole so if we try to create triggers on an existing bucket we get an error when SLS tries to create IamRoleCustomResourcesLambdaExecution.

It looks like this role is created as the lambda execution role for a custom lambda that adds the event trigger to S3. It is always created if you’re using an existing bucket. See here and here.

There’s an analogous feature in the console where it grant S3 the permissions to invoke the bucket.

We solved this problem by manually adding the event triggers after deploying the lambda.

If you are looking for future features it would be nice if there was a way to specify the execution role for the custom lambda.

Hi @medikoo,

But the truth is that the doploymentrole has the trust permissions

the problem is when I remove “existing: true” in serverless.yml, cloudformation try to create resource s3 that exists, right now and when I use “existing: true”, cloudformation fails giving the above mentioned error

Did you manage to solve this? I still have this issue

I am getting an error when existing:true creating the internal Iam role, as all roles created in my aws account should have the permission boundary defined. An error occurred: IamRoleCustomResourcesLambdaExecution - API: iam:CreateRole User: arn:aws:sts::xxx is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::yyy:role/xxx-IamRoleCustomResourcesLa-ZZZZ

Did anyone find a solution?

My apologies, figured out my issue was our CI server had an old version of serverless on it before the existing property was supported. It works as expected.