aws-cdk: [rds] Database Secret broken after upgrade to 1.67

After upgrading a stack from CDK 1.60 to 1.67 not only the password was regenerated without changing the database password (see https://github.com/aws/aws-cdk/issues/10716), also the values for host, port, dbName are entirely missing. Our application cannot access the DB anymore as all connection data was pulled from Secret Manager.

It seems like the changed ExcludeCharacters value caused a regenerate of the secret.

Environment

  • CLI Version :
  • Framework Version: 1.67.0
  • Node.js Version: 14.11
  • OS : MacOS 10.15.7
  • Language (Version): TypeScript 4.0.3

This is 🐛 Bug Report

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 12
  • Comments: 34 (25 by maintainers)

Most upvoted comments

@jogold Great tip, thanks so much!

The following worked for me:

  1. Add the code mentioned by @jogold to my construct. Deploy.
  2. Copy the newly generated secret’s value and save it as the value of the old DB secret.
  3. Remove the above-mentioned code from my construct. Deploy.

This is how you can unblock yourself and upgrade (database is a rds.DatabaseInstance or rds.DatabaseCluster in the code below).

// Create a new DB secret, use the same username as the one currently used
const dbSecret = new rds.DatabaseSecret(this, 'Secret', { username: '<your master username here>' });

// Override MasterUserPassword property in AWS::RDS::DBInstance or AWS::RDS::DBCluster
const cfnDatabase = database.node.defaultChild as cdk.CfnResource;
cfnDatabase.addPropertyOverride('MasterUserPassword', dbSecret.secretValueFromJson('password').toString());

// Override SecretId property in the AWS::SecretsManager::SecretTargetAttachment
const attachment = database.secret?.node.defaultChild as cdk.CfnResource;
attachment.addPropertyOverride('SecretId', dbSecret.secretArn);

After this, if you need to reference your secret elsewhere in your code you can still use database.secret as it references the attachment and not the secret itself.

(your old secret remains in place and linked to your instance/cluster but for the username only: its password is now useless, not in sync with your database anymore)

Ok, thanks for the explanation.

I understand the appeal of changing it so that this doesn’t happen in the future, but how can people unblock themselves with the current situation? Would deleting the database, and then restoring it from the snapshot be the only viable way?

This is the critical point for us right now - We’re currently now stuck on an old version of CDK because we cannot upgrade without breaking multiple clusters simultaneously. We have a large number of CDK provisioned databases and fixing them all manually in some way is obviously less than ideal 😃

I’m happy to perform a fairly involved workaround as a once-off if we can devise one/to get us to a better place but haven’t had time to devise/test much in this direction yet. It’d be ideal if the workaround could be implemented “in-CDK” so to speak rather than having to fiddle around externally (ie: direct AWS API manipulation/etc).