terraform-provider-aws: [Bug]: r/aws_db_instance_automated_backups_replication: `unexpected state 'Pending'`

Terraform Core Version

1.5.3

AWS Provider Version

5.8.0

Affected Resource(s)

aws_db_instance_automated_backups_replication

Expected Behavior

Creating the db_instance_automated_backups_replication resource should complete without any error

Actual Behavior

After creating a primary instance from this module; when creating db_instance_automated_backups_replication resource, it errors out.

Relevant Error/Panic Output Snippet

Error: waiting for DB instance automated backup (arn:aws:rds:us-west-1:<accountnumber>:auto-backup:ab-<string>) create: unexpected state 'Pending', wanted target 'replicating'. last error: %!s(<nil>)

Terraform Configuration Files

provider "aws" {
    profile = "default"
    region = "us-west-2"
    assume_role {
        role_arn     = "<arn of assumed role>"
        session_name = "TerraformAdminSession"
    }

}

provider "aws" {
    profile = "default"
    region = "us-west-1"
    assume_role {
        role_arn     = "<arn of assumed role>"
        session_name = "TerraformAdminSession"
    }
    alias = "cross_replica"
}

module "master" {
    source  = "terraform-aws-modules/rds/aws"
    version = "6.1.0"

  identifier = "test-master"

  engine               = "postgres"
  engine_version       = "14.6"
  family               = "postgres14"
  instance_class       = "db.m5.xlarge"

  allocated_storage     = "2000"
  max_allocated_storage = 3000
  db_name  = "replicaPostgresql"
  username = "replica_postgresql"
  port     = "5432"

  multi_az               = false

  maintenance_window              = "Mon:00:00-Mon:03:00"
  backup_window                   = "03:00-06:00"
  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade"]

  backup_retention_period = 1
  skip_final_snapshot     = true
  deletion_protection     = false

}

resource "aws_kms_key" "default" {
  description = "Encryption key for automated backups"

  provider = aws.cross_replica
}

resource "aws_db_instance_automated_backups_replication" "cross_region_replication" {
    depends_on = [ module.master ]
    source_db_instance_arn = module.master.db_instance_arn
    kms_key_id             = aws_kms_key.default.arn
    
    provider = aws.cross_replica
}

Steps to Reproduce

Need to have your ~/.aws/credentials with the appropriate access key to assume a role to the account with the [default] profile
Run terraform init and apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

About this issue

Original URL
State: closed
Created a year ago
Reactions: 16
Comments: 25 (2 by maintainers)

Most upvoted comments

I am working on a fix…

+21

ewbankkit on Jul 31, 2023

I’m having the exact same issue, with a slightly different setup. My DB already exists in us-east-1, and I’m adding backup replication to us-west-1.

Terraform v1.5.3 on linux_amd64

provider registry.terraform.io/hashicorp/aws v5.8.0 I’ve also tested with versions 4.9.0 and 4.23.0.

resource "aws_db_instance_automated_backups_replication" "dr_default" {
  source_db_instance_arn = var.source_db_instance_arn
  retention_period       = 1
  kms_key_id             = "arn:aws:kms:us-west-1:01234567890:key/abcd123456"
}

When I apply, it shows only the replication to be created.

  # module.replication.aws_db_instance_automated_backups_replication.dr_default will be created
  + resource "aws_db_instance_automated_backups_replication" "dr_default" {

But then I get the same error output, with a tainted status, and also the replication is eventually successful.

I can untaint the replication resource, and everything seems fine.

A similar workflow occurs on import->update. As a sanity check, I setup cross region snapshot replication manually in console and then I was able to import successfully (with key arn:aws:kms:us-west-1:01234567890:key/abcd123456). After that, on apply there were no changes, so everything is fine. However, if I then change the retention, which would force a recreate, the apply fails after destroying and then attempting to create.

 unexpected state 'Pending', wanted target 'replicating'.

Update to add one more behavior. After an untaint, if I remove the aws_db_instance_automated_backups_replication from configuration and apply, sometimes, but not every time, I’ll have this error:

aws_db_instance_automated_backups_replication.dr_default: Destroying... [id=arn:aws:rds:us-west-1:01234567890:auto-backup:ab-abcd123456]
╷
│ Error: error stopping RDS instance automated backups replication (arn:aws:rds:us-west-1:01234567890:auto-backup:ab-abcd123456): 
  InvalidDBInstanceState: DB Instance arn:aws:rds:us-east-1:01234567890:db:drtest is not replicating to the current region.
│       status code: 400, request id: 988bf200-7e95-414c-9fea-da4f9d1c6de8

taylorbartley on Jul 20, 2023

Although there is a workaround, we would still want a fix for this, please.

ixl-cchew on Jul 25, 2023

I was able to resolve temporarily by removing the state and importing it back. Since the resource is already replicating, plan shows no changes afterwards

So you did a terraform state rm [module].[resourcename] and then just apply it again?

terraform state rm [module].[resourcename], then terraform import [module].[resourcename] [resource id] You can get the id from the console since the resource has already been created

al-nwokedike-imprivata on Jul 23, 2023

I’m facing the same issue with 5.6.2.

ZakariaHili on Jul 20, 2023

I am also experiencing a similar issue with the following error: Error: waiting for DB instance automated backup (arn:aws:rds:??:??:auto-backup:ab-??) create: unexpected state 'Pending', wanted target 'replicating'. last error: %!s(<nil>)

al-nwokedike-imprivata on Jul 20, 2023

@ewbankkit - will this fix be propagated to 4.x versions?

sportymsk on Aug 1, 2023

It looks like the AWS RDS API status codes change capitalization without any notification 😢. We can reproduce this in our CI testing.

ewbankkit on Jul 31, 2023

I was able to resolve temporarily by removing the state and importing it back. Since the resource is already replicating, plan shows no changes afterwards

So you did a terraform state rm [module].[resourcename] and then just apply it again?

terraform state rm [module].[resourcename], then terraform import [module].[resourcename] [resource id] You can get the id from the console since the resource has already been created

The following also works as a workaround :

terraform untaint aws_db_instance_automated_backups_replication.your_resource_name

toadjaune on Aug 1, 2023

I was able to resolve temporarily by removing the state and importing it back. Since the resource is already replicating, plan shows no changes afterwards

So you did a terraform state rm [module].[resourcename] and then just apply it again?

terraform state rm [module].[resourcename], then terraform import [module].[resourcename] [resource id] You can get the id from the console since the resource has already been created

This worked! Thank you! And also just for clarity sake, it seems like the [resource id] is simply the ARN of the backup.

ixl-cchew on Jul 24, 2023

I was able to resolve temporarily by removing the state and importing it back. Since the resource is already replicating, plan shows no changes afterwards

So you did a terraform state rm [module].[resourcename] and then just apply it again?

ixl-cchew on Jul 21, 2023

I was able to resolve temporarily by removing the state and importing it back. Since the resource is already replicating, plan shows no changes afterwards

al-nwokedike-imprivata on Jul 20, 2023