serverless: First time deploy to aws apigateway errors out with: Invalid tag key. Tag keys cannot start with 'aws:' (case insensitive)

First time deploy in aws api gateway fails creation with tagKey error on cloudformation aws: tags

When running serverless for the first time the deployment is failing when serverless is trying to reconcile tags on api gateway and it sees the AWS newly added aws:cloudformation:logical-id, aws:cloudformation:stage, aws:cloudformation:stack-name and then tries to remove them. However, they are tags that are set by cloudformation and they cannot be added or removed outside of aws.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-resource-tags.html

Traced code to https://github.com/serverless/serverless/blob/c5ba682a6bc4fc96151c75cdf50cff2468d6def5/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js#L319

Basically it needs to skip trying to remove any tag that starts with ‘aws:’

serverless.yml

provider:
  name: aws
  runtime: python3.7
  stage: ${env:LABEL,'local'}
  versionFunctions: false
  memorySize: 512

stackTags:
    something: "no"

  customDomain:
    endpointType: regional
    domainName: "${ssm:/${env:LABEL}/infrastructure/something/domain_name~true}"
    stage: "${self:provider.stage}"
    certificateArn: "${ssm:/${env:LABEL}/infrastructure/dns/acm_wildcard_certificate/arn~true}"
    securityPolicy: tls_1_2

  stageSettings:
    MethodSettings:
      DataTraceEnabled: true
      HttpMethod: "*"
      LoggingLevel: "INFO"
      ResourcePath: "/*"
      MetricsEnabled: true

LABEL=blah sls deploy output

...
  Serverless Error ---------------------------------------
 
  Invalid tag key. Tag keys cannot start with 'aws:' (case insensitive)

1.83.2 and also exists with latest 2.x

Serverless: Setting custom naming conventions...
Serverless: Setting custom function names...Framework Core: 1.83.2 (local)Plugin: 3.8.4
SDK: 2.3.2
Components: 2.34.9

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (2 by maintainers)

Most upvoted comments

We are dealing the same issue (talking with the AWS folks). Our pipeline was broken. After digging thru the problem, it seems to us that AWS adding three additional aws tags in the ‘APIGateway:stage’. You can use this sample cloudformation template to re-create the problem:

Description: >-
  AWS CloudFormation to show a problem with tags
Resources:
  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: kel2
      EndpointConfiguration:
        Types:
        - REGIONAL
      Policy: ''
  ApiGatewayDeployment:
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId:
        Ref: ApiGatewayRestApi
      StageName: __unused__stage__
    DependsOn:
    - ApiGatewayMethod
  ApiGatewayMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      AuthorizationType: NONE
      HttpMethod: GET
      ResourceId:
        Ref: ApiGatewayResourceV1
      RestApiId:
        Ref: ApiGatewayRestApi
      Integration:
        Type: MOCK
      MethodResponses: []
  ApiGatewayResourceV1:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId:
        Fn::GetAtt:
        - ApiGatewayRestApi
        - RootResourceId
      PathPart: v1
      RestApiId:
        Ref: ApiGatewayRestApi
  ApiGatewayStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      DeploymentId:
        Ref: ApiGatewayDeployment
      RestApiId:
        Ref: ApiGatewayRestApi
      StageName: kel2
      MethodSettings:
      - HttpMethod: "*"
        LoggingLevel: INFO
        MetricsEnabled: true
        ResourcePath: "/*"

After you run the cloudformation, two stages are created ‘kel2’ and ‘unused__stage’. Notice that kel2 will have three extra ‘aws:’ tags while unused__stage does not have it

Here are two workaround for us:

remove the tags from serverless.yml (which is undesirable)
Change the code under node_modules/serverless/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js (line 334) from: currentKey => typeof tagsMerged[currentKey] !== 'string' to: currentKey => !currentKey.startsWith('aws:') && typeof tagsMerged[currentKey] !== 'string'

kelvinatorHK on Apr 2, 2021

@pgrzesik Just FYI, I took this snapshot last week (when the problem occurred)

kelvinatorHK on Apr 6, 2021