serverless: serverless deploy fails after adding properties to a lambda event stream

This is a Bug Report

Description

I have a serverless.yml file which defines a lambda function which fires based on kinesis stream events. This would successfully deploy when running sls deploy. An example snippet of this is shown below.

...
functions:
  whos-got-the-func:
    handler: handler.WhosGotTheFunc
    timeout: 60
    environment:
      ...
    events:
      - stream:
          type: kinesis
          arn:
            Fn::ImportValue: ${env:KINESIS_FUNC_UPDATE}
...

If i then add some additional properties to the kinesis stream event, such as a batchSize, startingPosition and enabled then the sls deploy fails. Example of the changes shown below:

...
functions:
  whos-got-the-func:
    handler: handler.WhosGotTheFunc
    timeout: 60
    environment:
      ...
    events:
      - stream:
          type: kinesis
          arn:
            Fn::ImportValue: ${env:KINESIS_FUNC_UPDATE}
          batchSize: 100
          startingPosition: LATEST
          enabled: true
...

The error message output is as follows:

11:54:20.545 sls deploy -v
11:54:21.828 
11:54:21.846 Serverless: Updating Log Forwarding Resources...
11:54:21.847 Serverless: Log Forwarding Resources Updated
11:54:21.853 Serverless: Packaging service...
11:54:22.695 Serverless: Uploading CloudFormation file to S3...
11:54:22.810 Serverless: Uploading artifacts...
11:54:23.427 Serverless: Validating template...
11:54:24.206 Serverless: Updating Stack...
11:54:25.480 Serverless: Checking Stack update progress...
11:54:30.742 CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - MyCloudStack
11:55:02.387 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - whos-got-the-func
11:55:02.387 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - i-got-the-func
11:55:02.387 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - you-got-the-func
11:55:02.387 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - i-got-the-func
11:55:02.387 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - whos-got-the-func
11:55:02.387 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - we-got-the-func
11:55:02.387 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - shes-got-the-func
11:55:02.387 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - you-got-the-func
11:55:02.388 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - we-got-the-func
11:55:02.388 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - shes-got-the-func
11:55:07.633 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::EventSourceMapping - WhosGotTheFuncEvenSourceMappingKinesisABCDUpdateKinesisArn
11:55:07.633 CloudFormation - UPDATE_FAILED - AWS::Lambda::EventSourceMapping - WhosGotTheFuncEvenSourceMappingKinesisABCDUpdateKinesisArn
11:55:12.857 CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - MyCloudStack
11:55:18.107 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - i-got-the-func
11:55:18.107 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - whos-got-the-func
11:55:18.107 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - shes-got-the-func
11:55:18.108 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - we-got-the-func
11:55:18.108 CloudFormation - UPDATE_IN_PROGRESS - AWS::Lambda::Function - you-got-the-func
11:55:18.108 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - whos-got-the-func
11:55:18.108 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - i-got-the-func
11:55:18.108 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - shes-got-the-func
11:55:18.108 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - you-got-the-func
11:55:18.108 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::Function - we-got-the-func
11:55:18.108 CloudFormation - UPDATE_COMPLETE - AWS::Lambda::EventSourceMapping - WhosGotTheFuncEvenSourceMappingKinesisABCDUpdateKinesisArn
11:55:28.582 CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - MyCloudStack
11:55:28.582 CloudFormation - DELETE_COMPLETE - AWS::Lambda::EventSourceMapping - WhosGotTheFuncEvenSourceMappingKinesisABCDUpdateKinesisArn
11:55:28.582 CloudFormation - UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - MyCloudStack
11:55:28.583 Serverless: Operation failed!
11:55:28.587 
11:55:28.588   Serverless Error ---------------------------------------
11:55:28.588 
11:55:28.588   An error occurred: WhosGotTheFuncEvenSourceMappingKinesisABCDUpdateKinesisArn - 
The event source arn (arn:aws:kinesis:ap-southeast-2:6969696969:stream/Update-Kinesis)  
and function (MyCloudStack-whos-got-the-func) provided mapping already exists. 
Please update or delete the existing mapping with UUID 666666-7777-8888-9999-000000000000.
11:55:28.588 
11:55:28.588   Get Support --------------------------------------------
11:55:28.588      Docs:          docs.serverless.com
11:55:28.589      Bugs:          github.com/serverless/serverless/issues
11:55:28.589      Forums:        forum.serverless.com
11:55:28.589      Chat:          gitter.im/serverless/serverless
11:55:28.589 
11:55:28.589   Your Environment Information -----------------------------
11:55:28.589      OS:                     linux
11:55:28.589      Node Version:           6.11.3
11:55:28.589      Serverless Version:     1.22.0
11:55:28.589 
11:55:28.596 make: *** [_deploy] Error 1
11:55:30.069 make: *** [deploy] Error 2
11:55:30.078 [go] Current job status: failed.

To solve this problem you need to manually remove the existing/previous trigger from the lambda function via the console. However, the behaviour i would expect here is that the deploy succeeds overriding the initial trigger but just defining the additional properties. Ie. no manual intervention required.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 27
  • Comments: 34 (2 by maintainers)

Most upvoted comments

Well… that’s a bit awkard…

My function was deleted but the mapping between it and the stream wasn’t so… I need to create a new one manually, magically the mapping appears in the new function, delete the mapping and the delete the function again,

hope this helps someone

+33
OscarGalindo on Sep 17, 2018

Closing since this issue is quite stale and we’ve pushed some changes in the past. Furthermore one can find some workarounds in this thread.

Feel free to conitnue the discussion…

+30
pmuens on Feb 6, 2019

Can we reopen this? It’s makes the event mapping unusable via Serverless.com.

+24
stevesuh on May 21, 2019

Since this issue still exists, I appreciate the (manual) workaround, but long-term requiring manual effort for CloudFormation and Serverless templates is anti-thetical to the purporse of these services. Not addressing this customer impacting root cause really is not acceptable (not sure why this was ever closed). Please reopen.

+17
RampantLions on Aug 7, 2019

Please consider reopening. Manual or even scripted workarounds make serverless much less of an interesting proposition for managing lambda infrastructure. If you handle creation of mappings via config, you automatically create the expectation of being able to handle the updates, too.

+15
nocon on Jul 5, 2019

Possible work-around through aws-cli: aws lambda delete-event-source-mapping --uuid <UUID from error message>

+14
apoorvavinod on Apr 30, 2019

Why closed? This is still an issue

+9
marksiano on Jul 21, 2020

This problem is making it hard for us to deploy code into our production environment - would be great if this issue were re-opened!

+9
TABTheoBreuerWeil on Dec 17, 2019

The expected behavior is that serverless corrects the erroneous binding (removes then readds). Serverless should remain idempotent and simply create or update the stack as it’s defined in serverless.yml without error.

This is pretty easy to reproduce, just deploy a simple example like https://github.com/DavidWells/serverless-workshop/tree/master/lessons-code-complete/events/kinesis/kinesis-basic and then go to CloudFormation and delete the stack. Make sure it’s deleted and then re-deploy. It errors 100% of the time for me.

+9
ian on Apr 1, 2019

The workaround of creating a new function with the same name just to delete the mapping did work for me too. But it would indeed be very good if serverless did the right thing in the first place.

+9
evanderkoogh on Oct 11, 2018

Another workaround is to delete the mapping in the command line via the aws-cli. This worked for me:

aws lambda delete-event-source-mapping --uuid abc123-def456-xyz-987-11111EXAMPLE

Docs: https://docs.aws.amazon.com/cli/latest/reference/lambda/delete-event-source-mapping.html

Make sure you’ve got your credentials set up to use the aws-cli for your OS: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

+4
JessicaBarclay on May 1, 2020

Leaving breadcrumbs 😃

I’ve seen this issue when there was no actual change to the event source mapping, but the syntax of the template changed around the use of Fn::GetAtt from the short-form with dot-notation to the long form with array notation. This was for an SQS event source mapping.

+1
dls314 on Oct 27, 2020

This doesn’t seem to be a serverless issue as much as an issue with CloudFormation. It appears that serverless is generating the EventSourceMapping correctly according to the Cloudformation docs:

"MyLambdaFunctionEventSourceMappingDynamodbMyTable": {
  "Type": "AWS::Lambda::EventSourceMapping",
  "DependsOn": "IamRoleLambdaExecution",
  "Properties": {
    "BatchSize": 50,
    "EventSourceArn": {
      "Fn::GetAtt": [
        "MyTable",
        "StreamArn"
      ]
    },
    "FunctionName": {
      "Fn::GetAtt": [
        "MyLambdaFunction",
        "Arn"
      ]
    },
    "StartingPosition": "LATEST",
    "Enabled": "True",
    "MaximumBatchingWindowInSeconds": 60,
    "MaximumRetryAttempts": 2
  }
}

This issue also mentions that this is a known issue with cloudformation currently.

+1
Aposhian on Apr 9, 2020

FYI we solved this by being more diligent in specifying the “DependsOn” field where appropriate. After establishing proper relationships between our resources, this problem never appeared again.

@nojohnny101 Could you be a little more specific about what you did? It appears that Serverless declares the EventSourceMapping to depend on the IamRoleLambdaExecution only.

+1
Aposhian on Apr 9, 2020

please reopen

+1
bradacina on Jul 23, 2019

Confirmed this is still happening in 1.40.0, please reopen.

+1
ian on Apr 1, 2019
Read more comments on GitHub
← serverless: Serverless Deploy Error: Path must be a string. Received undefined
serverless: Serverless Error - Function not found... →