serverless-application-model: S3 Event triggers not working

@phongtran227 @pierremarieB

You can add a AWS::Lambda::Permission to your Resources. It should look something like the following:

  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !GetAtt MyFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: 's3.amazonaws.com'
      SourceAccount: !Sub ${AWS::AccountId}
      SourceArn: !GetAtt MyBucket.Arn

That worked for me.

If you use the S3 event in SAM, then you don’t see any trigger in the Lambda configuration panel. But the Lambda function is executed when you drop a file in the S3 bucket.

Just wondering why this is closed? I’m seeing the same issue. The bucket event source does not show up in the console. It makes it very confusing to know what is going on.

For those that are using the s3 template created by the cli, change the template for this one and the s3 trigger will be created. I also deleted the stack created by the old template and recreated the stack with the new one. Thanks @henrikbjorn

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  s3-lambda

Parameters:
  AppBucketName:
    Type: String
    Description: "REQUIRED: Unique S3 bucket name to use for the app."

Resources:
  S3JsonLoggerFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handlers/s3-json-logger.s3JsonLoggerHandler
      Runtime: nodejs12.x
      MemorySize: 128
      Timeout: 60
      Policies:
        S3ReadPolicy:
          BucketName: !Ref AppBucketName
      Events:
        S3NewObjectEvent:
          Type: S3
          Properties:
            Bucket: !Ref AppBucket
            Events: s3:ObjectCreated:*
            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: ".json"
                    
  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !GetAtt S3JsonLoggerFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: 's3.amazonaws.com'
      SourceAccount: !Sub ${AWS::AccountId}
      SourceArn: !GetAtt AppBucket.Arn

  AppBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref AppBucketName

The S3 triggers are working but they are not appearing in the lambda console.

This is a breaking change, so we wouldn’t be able to do this unless we made a new version of SAM. I agree, though, that this should be fixed.

Lu’s comment (https://github.com/awslabs/serverless-application-model/issues/300#issuecomment-408950770) describes what needs to be changed.

You can add a AWS::Lambda::Permission to your Resources. It should look something like the following:

  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !GetAtt MyFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: 's3.amazonaws.com'
      SourceAccount: !Sub ${AWS::AccountId}
      SourceArn: !GetAtt MyBucket.Arn

That worked for me.

Thank you.

There should be a note about this in the official docs. Just sent a feedback to them.

If you use the S3 event in SAM, then you don’t see any trigger in the Lambda configuration panel. But the Lambda function is executed when you drop a file in the S3 bucket.

I get this problem too, when I deploy app to the aws, I config the s3 trigger, but I don’t see it on the aws console, I also tried the dynamodb or api gateway trigger and they can show on the aws console. I don’t know why?

Another question is I can’t set the bucket name when I config s3 event; I find the older doc config they can set the s3 bucket name. Is there something different between the old and now version ?

For those that are using the s3 template created by the cli, change the template for this one and the s3 trigger will be created. I also deleted the stack created by the old template and recreated the stack with the new one. Thanks @henrikbjorn

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  s3-lambda

Parameters:
  AppBucketName:
    Type: String
    Description: "REQUIRED: Unique S3 bucket name to use for the app."

Resources:
  S3JsonLoggerFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handlers/s3-json-logger.s3JsonLoggerHandler
      Runtime: nodejs12.x
      MemorySize: 128
      Timeout: 60
      Policies:
        S3ReadPolicy:
          BucketName: !Ref AppBucketName
      Events:
        S3NewObjectEvent:
          Type: S3
          Properties:
            Bucket: !Ref AppBucket
            Events: s3:ObjectCreated:*
            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: ".json"
                    
  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !GetAtt S3JsonLoggerFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: 's3.amazonaws.com'
      SourceAccount: !Sub ${AWS::AccountId}
      SourceArn: !GetAtt AppBucket.Arn

  AppBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref AppBucketName

This doesn’t work for me because it complains about circular dependencies:

E3004: Circular Dependencies for resource DynamoS3Function. Circular dependency with

Used the provided template to reproduce the issue.

The Lambda permissions created by SAM looks like this:

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "LogToWatchS3CreateObjectPermission-F3WNVM1OUAMX",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:invokeFunction",
      "Resource": "arn:aws:lambda:<region>:<account_id>:function:LogToWatch-1WQI7PUOH1EF",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "<account_id>"
        }
      }
    }
]
}

But Lambda expects this in order to show the trigger in the console:

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "lambda-a5c4fbbd-61fc-4b08-82b4-7dd593c7f65f",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-1:303769779339:function:test-LogToWatch-1WQI7PUOH1EF",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "303769779339"
        },
        "ArnLike": {
          "AWS:SourceArn": "<S3 bucket Arn>"
        }
      }
    }
  ]
}

It is missing this part:

"ArnLike": {
          "AWS:SourceArn": "<S3 bucket Arn>"
}

In the cloudformation template, the AWS::Lambda::Permission resource after transformed looks like this:

"LogToWatchS3CreateObjectPermission": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "Action": "lambda:invokeFunction",
        "SourceAccount": {
          "Ref": "AWS::AccountId"
        },
        "FunctionName": {
          "Ref": "LogToWatch"
        },
        "Principal": "s3.amazonaws.com"
      }
    }

It is missing the SourceArn property. It should be something like this:

"LogToWatchS3CreateObjectPermission": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "Action": "lambda:invokeFunction",
        "SourceAccount": {
          "Ref": "AWS::AccountId"
        },
        "FunctionName": {
          "Ref": "LogToWatch"
        },
        "Principal": "s3.amazonaws.com",
        "SourceArn": {
             "Fn::GetAtt": "TargetBucket.Arn"
         }
      }
    }

However the issue is that, currently SAM makes the permission not scoped to a specific bucket. If we are to fix it in SAM making it scoped to a specific bucket so that it can show up properly in Lambda console, this could potentially break existing customer who expects broader permissions.

Though we are not changing the existing behaviour, there is a simple way to deal with the problem gracefully.

First, let me briefly explain why we have not modified the existing Lambda resource policy and are not planning to do so. As mentioned above, the problem with Console not showing the trigger comes from the fact that the resource policy which is created by SAM on S3 Event generation does not restrict Lambda access to a single bucket. If we change the policy now, it will break working code for the customers who already rely on broader permissions, as mentioned in the referenced explanation and here.

Second, I’d like to recommend the way to narrow down the permissions so they will work with Console, avoid the Circular Dependency pitfall, and reduce the boilerplate. It is based on the ideas many of you have already figured out and it leverages SAM Connector resource we have recently introduced.

1. Instead of crafting AWS::Lambda::Permission use AWS::Serverless::Connector which we introduced in September 2022. You can read more on Connectors here

2. To guarantee that there is no circular dependency, hardcode your bucket name.

Here is an example, based on the one which was submitted when the issue was open

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Description: Example

Resources:
  LogToWatch:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs16.x
      Handler: index.handler
      InlineCode: |
        exports.handler = async (event) => {
          console.log(event);
        };
      Timeout: 300
      Policies: AmazonS3ReadOnlyAccess
      Events:
        S3CreateObject:
          Type: S3
          Properties:
            Bucket: 
              Ref: TargetBucket
            Events: s3:ObjectCreated:Put
  TargetBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: target-bucket-for-test-102546
  MyConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Type: AWS::S3::Bucket
        Arn: !Sub arn:${AWS::Partition}:s3:::target-bucket-for-test-102546
      Destination:
        Id: LogToWatch
      Permissions:
        - Write

Notice how we have to set bucket name explicitly

Properties:
    BucketName: target-bucket-for-test-102546

And then reference Connector source by the ARN

Source:
    Type: AWS::S3::Bucket
    Arn: !Sub arn:${AWS::Partition}:s3:::target-bucket-for-test-102546

P.S If you don’t have to stick to AmazonS3ReadOnlyAccess for the compatibility reasons, you can use another connector instead of it.

MyConnector2:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: LogToWatch
      Destination:
        Type: AWS::S3::Bucket
        Arn: !Sub arn:${AWS::Partition}:s3:::target-bucket-for-test-102546
      Permissions:
        - Read

Notice that Source and Destination have exchanged their places and we require Read permissions for Lambda to S3 access.

A connector will result in a more granular policy generated. Compare this one generated by connector:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectLegalHold",
                "s3:GetObjectRetention",
                "s3:GetObjectTorrent",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionTorrent",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::target-bucket-for-test-102546",
                "arn:aws:s3:::target-bucket-for-test-102546/*"
            ],
            "Effect": "Allow"
        }
    ]
}

To the one from AmazonS3ReadOnlyAccess:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "*"
        }
    ]
}

https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/79

@moosakhalid You also need access to the S3 bucket. I found that it is important to specify Version when specifying policies.

I have only tried where i created the s3 bucket in the same CloudFormation template.

i am experiencing similar issue with SQS events. using SAM to deploy lambda with SQS event the lambda receives messages from the queue but the trigger is not visible via AWS console.

However the issue is that, currently SAM makes the permission not scoped to a specific bucket. If we are to fix it in SAM making it scoped to a specific bucket so that it can show up properly in Lambda console, this could potentially break existing customer who expects broader permissions.

please note that when using the Events you are expecting SPECIFIC permissions. giving broader permissions is an issue not a feature. when creating a lambda using SAM and giving an S3 as a trigger i expect that only that S3 is able to trigger the lambda. giving broader permissions than that seems unsecured.

I’m seeing the same behavior as @bottemav