aws-cdk: (dynamo-db): Failures with Adding GSI with auto-scaling to a replicated DynamoDB table via CDK

What is the problem?

We are using CDK to manage our Dynamo tables. In alpha, the tables are not replicated globally, but in beta/gamma/prod there is replication from us-east-1 to us-west-2 and eu-west-1.

We have an existing table called “Promotions” that is already deployed and is replicating to these regions. It also already has an existing GSI. We want to add a new GSI to this table. It successfully deploys in alpha where it is NOT being replicated. But when it tries to deploy to Beta where it is being replicated, the deployment fails with:

"table/****/index/*****|dynamodb:index:ReadCapacityUnits|dynamodb already exists"

After the initial failure, we deleted the GSI manually from the AWS console and also ran AWS CLI commands to delete these dynamodb:index:ReadCapacityUnits and dynamodb:index:WriteCapacityUnits resources from all 3 relevant regions, but we see the exact same failure when trying to redeploy.

Is there something inherently wrong with trying to add a new auto-scaling GSI to a table with replication regions?

Reproduction Steps

EXPERIMENT 1: Add the GSI with auto-scaling - FAILED

  1. Global table deployed
  2. Add GSI with auto-scaling - FAILED with policy resource conflict (probably a bug in CDK)

EXPERIMENT 2: Add the GSI with auto-scaling and another GSI without auto-scaling - FAILED

  1. Global table deployed with GSI with auto-scaling
  2. Add GSI with auto-scaling - FAILED with policy resource conflict (probably a bug in CDK)

EXPERIMENT 3: Add the GSI without auto-scaling - FAILED

  1. Global table deployed with GSI without auto-scaling - FAILED because gsi needs auto-scaling (seems legit)

EXPERIMENT 4: Add the table alone, and then the GSI without auto-scaling, and then modify to use auto-scaling - FAILED

  1. Global table deployed
  2. Add GSI without auto-scaling - GSI inherits table’s auto scaling
  3. Add auto-scaling to GSI from step 2 - FAILED with policy resource conflict (seems legit because 2nd step created it behind the scenes) a. I can’t just skip this step though, because if a dev stack tries to deploy at step 2 it’ll fail deployment because gsi needs auto-scaling b. It also seems like this auto-scaling policy will need to be orphaned forever, since this step proves I can never reclaim it

EXPERIMENT 5: Add the table with a GSI with auto-scaling, and then another GSI without auto-scaling, and then modify to use auto-scaling - FAILED

  1. Global table deployed with GSI with auto-scaling
  2. Add GSI without auto-scaling - GSI inherits table’s auto scaling
  3. Add auto-scaling to GSI from step 2 - FAILED with policy resource conflict (seems legit because 2nd step created it behind the scenes) a. I can’t just skip this step though, because if a dev stack tries to deploy at step 2 it’ll fail deployment because gsi needs auto-scaling b. It also seems like this auto-scaling policy will need to be orphaned forever, since this step proves I can never reclaim it

What did you expect to happen?

The new GSI to be deployed with its own auto-scaling settings

What actually happened?

Deployment failed

CDK CLI Version

2.13.0

Framework Version

No response

Node.js Version

17

OS

MacOS

Language

Python

Language Version

No response

Other information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (10 by maintainers)

Most upvoted comments

Will follow up further with AWS support, thanks for investigating