serverless: DDB Stream Lambda creation fails due to IAM policy not being updated first

This is a Bug Report

Description

What went wrong?

While updating an existing deployment with a new DDB Stream lambda, while simultaneously updating the IAM policy for the role and the DDB table to add the stream, the AWS::Lambda::EventSourceMapping fails to create with the following error:

Cannot access stream arn:aws:dynamodb:us-west-2:123456789012:table/logger-api-gateway-stage-2/stream/2017-07-26T18:04:04.509. Please ensure the role can perform the GetRecords, GetShardIterator, DescribeStream, and ListStreams Actions on your stream in IAM.

These permissions are due to be added in the update, but because the AWS::Lambda::EventSourceMapping is not set to depend on the IAM role, the updates don’t happen in the correct sequence.

What did you expect should have happened?

AWS::Lambda::EventSourceMapping’s that are for DynamoDB Streams should have “DependsOn”: “IamRoleLambdaExecution”

What was the config you used?

  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:DescribeStream
        - dynamodb:GetItem
        - dynamodb:GetRecords
        - dynamodb:GetShardIterator
        - dynamodb:ListStreams
        - dynamodb:PutItem
        - dynamodb:Query
        - dynamodb:Scan
        - dynamodb:UpdateItem
      Resource:
        - Fn::GetAtt:
          - LogsTable
          - StreamArn
functions:
  bus:
    description: Processes changes from DynamoDB and relays as appropriate
    handler: logs/bus.process
    events:
      - stream:
          type: dynamodb
          arn:
            Fn::GetAtt:
              - LogsTable
              - StreamArn

What stacktrace or error message from your provider did you see?

  Serverless Error ---------------------------------------
 
  An error occurred while provisioning your stack: BusEventSourceMappingDynamodbLogsTable - Cannot access stream arn:aws:dynamodb:us-west-2:012345678901:table/logger-api-gateway-stage-2/stream/2017-07-26T18:04:04.509. Please ensure the role can perform the GetRecords, GetShardIterator, DescribeStream, and ListStreams Actions on your stream in IAM..

Additional Data

Serverless Framework Version you’re using:

1.16.1

Operating System:

ubuntu 16.04

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 6
Comments: 17 (7 by maintainers)

Most upvoted comments

Hi @flyinbutrs, I am experiencing the same problem with Kinesis Streams.

I get a similar error:

An error occurred: MyFunctionEventSourceMappingMyKinesisStream - Cannot access stream arn:aws:kinesis:us-east-1:XXX:stream/YYY. Please ensure the role can perform the GetRecords, GetShardIterator, DescribeStream, and ListStreams Actions on your stream in IAM.

Have you tested your suggested fix related to dependsOn?

CC @pmuens you can reproduce the bug by starting with a yaml configuration with no iamRoleStatements and no events, such as:

service: xxx
provider:
  name: aws
  runtime: python2.7
functions:
  myFunction:
    handler: handler.my_handler

You deploy it, and then you update the config to:

service: xxx
provider:
  name: aws
  runtime: python2.7
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - kinesis:GetRecords
        - kinesis:GetShardIterator
        - kinesis:DescribeStream
        - kinesis:ListStreams
      Resource:
        - Fn::GetAtt:
          - MyKinesisStream
          - Arn

functions:
  myFunction:
    handler: handler.my_handler
    events:
      - stream:
          type: kinesis
          batchSize: 100
          arn:
            Fn::GetAtt:
              - MyKinesisStream
              - Arn

resources:
  Resources:
    MyKinesisStream:
      Type: AWS::Kinesis::Stream
      Properties:
        ShardCount: 1

I believe this did not happen until a few months ago because AWS was not validating Lambda’s permissions when you created a new EventSourceMapping.

I wouldn’t consider the bug super critical, as everything works fine if you sls remove and then sls deploy again, but it can be quite frustrating for beginners.

alexcasalboni on Sep 19, 2017

Ran into the same problem. Not sure if this helps, but I think this is actually a problem with the error message— it doesn’t list all the permissions you actually need. I found at https://docs.amazonaws.cn/en_us/lambda/latest/dg/with-kinesis.html under “Execution role permissions” it says you need more permissions. I’ve updated my serverless.yml with the following and it seems to work:

      - Effect: "Allow"
        Action:
          - kinesis:DescribeStream
          - kinesis:DescribeStreamSummary
          - kinesis:GetRecords
          - kinesis:GetShardIterator
          - kinesis:SubscribeToShard
        Resource: your_stream_arn

      - Effect: "Allow"
        Action:
          - kinesis:ListStreams
          - kinesis:ListShards
        Resource: "*"

jheising on Jun 29, 2021

Seems like this was fixed on AWS end (see convo in #4427). Feel free to re-open if this is still an issue.

It is still an issue with dynamodb and Iam policy dependencies. How do I re-open this issue?

patelman on Jun 21, 2019

@flyinbutrs Update: I took an hour to work on it and opened this PR.

It works 😃

Unfortunately, I haven’t updated all the tests yet, so it won’t be merged asap. Hopefully someone can help with that (CC @pmuens @horike37), otherwise I’ll keep working on it next week.

In the meantime, you could test if it works for you as well.

alexcasalboni on Oct 31, 2017

I suspect that in this line, it needs to be IamRoleLambdaExecutionPolicy, rather than IamRoleLambdaExecution. That way if the policy is to be updated, it’ll get updated before the EventSourceMapping is created.

https://github.com/serverless/serverless/blob/aebce86ab20faaf95db32e037990de81f28e8dda/lib/plugins/aws/package/compile/events/stream/index.js#L122

flyinbutrs on Jul 26, 2017