aws-cdk: Cannot create Aurora Serverless cluster using RDS Construct

Is there any progress on this?

It would be really helpful to have an L2 construct for Aurora Serverless DB cluster including Data API, secret rotation, and methods to easily grant read/write access Lambdas or AppSync.

I set this up for our usage, it’s ugly, but it may be of use to someone who needs a “quick and dirty” version of Serverless Aurora:

Please ignore the lack of comments, and possibly broken code, I removed some proprietary information. 🤕

import {Connections, ISecurityGroup, IVpc, Port, SecurityGroup, SubnetSelection} from "@aws-cdk/aws-ec2";
import {
    CfnDBCluster,
    CfnDBSubnetGroup,
    DatabaseSecret,
    Endpoint,
    SecretRotation,
    SecretRotationApplication,
    SecretRotationOptions
} from "@aws-cdk/aws-rds";
import {AttachmentTargetType, ISecretAttachmentTarget, SecretAttachmentTargetProps, SecretTargetAttachment} from "@aws-cdk/aws-secretsmanager";
import {Construct, RemovalPolicy, Token} from "@aws-cdk/core";

export interface ServerlessAuroraProps {
    readonly vpc: IVpc;
    readonly subnets: SubnetSelection;

    readonly clusterName: string;

    readonly masterUsername?: string;
    readonly securityGroup?: ISecurityGroup;
    readonly secretRotationApplication?: SecretRotationApplication;

    readonly maxCapacity: number;
}

export class ServerlessAurora extends Construct implements ISecretAttachmentTarget {
    public securityGroupId: string;
    public clusterIdentifier: string;
    public clusterEndpoint: Endpoint;
    public secret: SecretTargetAttachment;
    public connections: Connections;
    public vpc: IVpc;
    public vpcSubnets: SubnetSelection;
    public secretRotationApplication: SecretRotationApplication;
    public securityGroup: ISecurityGroup;

    constructor(scope: Construct, id: string, private props: ServerlessAuroraProps) {
        super(scope, id);

        this.vpc = props.vpc;
        this.vpcSubnets = props.subnets;
        this.secretRotationApplication = props.secretRotationApplication || SecretRotationApplication.MYSQL_ROTATION_SINGLE_USER;

        const secret = new DatabaseSecret(this, "MasterUserSecret", {
            username: props.masterUsername || "root",
        });

        const securityGroup = props.securityGroup || new SecurityGroup(this, "DatabaseSecurityGroup", {
            allowAllOutbound: true,
            description: `DB Cluster (${props.clusterName}) security group`,
            vpc: props.vpc
        });
        this.securityGroup = securityGroup;
        this.securityGroupId = securityGroup.securityGroupId;

        const cluster = new CfnDBCluster(this, "DatabaseCluster", {
            engine: "aurora",
            engineMode: "serverless",
            engineVersion: "5.6",

            dbClusterIdentifier: props.clusterName,

            masterUsername: secret.secretValueFromJson("username").toString(),
            masterUserPassword: secret.secretValueFromJson("password").toString(),

            dbSubnetGroupName: new CfnDBSubnetGroup(this, "db-subnet-group", {
                dbSubnetGroupDescription: `${props.clusterName} database cluster subnet group`,
                subnetIds: props.vpc.selectSubnets(props.subnets).subnetIds
            }).ref,

            vpcSecurityGroupIds: [securityGroup.securityGroupId],

            storageEncrypted: true,

            // Maximum here is 35 days
            backupRetentionPeriod: 35,

            scalingConfiguration: {
                autoPause: true,
                secondsUntilAutoPause: 300,
                minCapacity: 1,
                maxCapacity: props.maxCapacity
            }
        });
        cluster.applyRemovalPolicy(RemovalPolicy.DESTROY, {applyToUpdateReplacePolicy: true});

        this.clusterIdentifier = cluster.ref;
        // create a number token that represents the port of the cluster
        const portAttribute = Token.asNumber(cluster.attrEndpointPort);
        this.clusterEndpoint = new Endpoint(cluster.attrEndpointAddress, portAttribute);

        if (secret) {
            this.secret = secret.addTargetAttachment('AttachedSecret', {target: this});
        }
        const defaultPort = Port.tcp(this.clusterEndpoint.port);
        this.connections = new Connections({securityGroups: [securityGroup], defaultPort});

        // This is currently causing errors when deploying, since it uses a SAM template under the hood
        // Error: "Received malformed response from transform AWS::Serverless-2016-10-31"
        // It also adds in a warning from the CDK:
        // "This stack is using the deprecated `templateOptions.transform` property. Consider switching to `templateOptions.transforms`."
        // Which has notified *all* of our DevOps when we upgraded.
        // this.addRotationSingleUser("Rotation");
    }

    /**
     * Adds the single user rotation of the master password to this cluster.
     */
    public addRotationSingleUser(id: string, options?: SecretRotationOptions): SecretRotation {
        if (!this.secret) {
            throw new Error('Cannot add single user rotation for a cluster without secret.');
        }
        return new SecretRotation(this, id, {
            secret: this.secret,
            application: this.secretRotationApplication,
            vpc: this.vpc,
            vpcSubnets: this.vpcSubnets,
            target: this,
            automaticallyAfter: options ? options.automaticallyAfter : undefined,
        });
    }

    public asSecretAttachmentTarget(): SecretAttachmentTargetProps {
        return {
            targetId: this.clusterIdentifier,
            targetType: AttachmentTargetType.CLUSTER
        };
    }
}

Working construct with CDK 1.55.0

import * as cdk from '@aws-cdk/core';
import * as rds from '@aws-cdk/aws-rds';
import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
import * as ec2 from '@aws-cdk/aws-ec2';
import * as kms from '@aws-cdk/aws-kms';

export interface AuroraServerlessProps {
  readonly engine: rds.IClusterEngine;
  readonly clusterIdentifier?: string;
  readonly parameterGroup?: rds.ParameterGroup;
  readonly vpc: ec2.IVpc;
  readonly vpcSubnets: ec2.SubnetSelection;
  readonly securityGroups?: ec2.ISecurityGroup[];
  readonly masterUsername?: string;
  readonly defaultDatabaseName?: string;
  readonly backup?: rds.BackupProps;
  readonly preferredMaintenanceWindow?: string;
  readonly storageEncryptionKey?: kms.IKey;
  readonly deletionProtection?: boolean;
  readonly removalPolicy?: cdk.RemovalPolicy;
  readonly scalingConfig?: rds.CfnDBCluster.ScalingConfigurationProperty;
  readonly enableDataApi?: boolean;
}

// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html
// https://madabout.cloud/2019/09/01/aws-data-api-for-amazon-aurora-serverless/
// https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-rds/lib/cluster.ts
// https://github.com/ysku/aurora-serverless-example/blob/master/lib/database.ts
export class AuroraServerless extends cdk.Resource implements ec2.IConnectable, secretsmanager.ISecretAttachmentTarget {
  /**
   * Identifier of the cluster
   */
  public readonly clusterIdentifier: string;

  /**
   * ARN of the cluster
   */
  public readonly clusterArn: string;

  /**
   * The endpoint to use for read/write operations
   */
  public readonly clusterEndpoint: rds.Endpoint;

  /**
   * Access to the network connections
   */
  public readonly connections: ec2.Connections;

  /**
   * Security group identifier of this database
   */
  public readonly securityGroups: ec2.ISecurityGroup[];

  /**
   * The secret attached to this cluster
   */
  public readonly secret?: secretsmanager.ISecret;

  private readonly secretRotationApplication: secretsmanager.SecretRotationApplication;

  /**
   * The VPC where the DB subnet group is created.
   */
  private readonly vpc: ec2.IVpc;

  /**
   * The subnets used by the DB subnet group.
   */
  private readonly vpcSubnets: ec2.SubnetSelection;

  constructor(scope: cdk.Construct, id: string, props: AuroraServerlessProps) {
    super(scope, id);

    const {
      clusterIdentifier,
      engine,
      parameterGroup,
      storageEncryptionKey,
      backup,
      preferredMaintenanceWindow,
      deletionProtection,
      removalPolicy,
      defaultDatabaseName,
      vpc,
      vpcSubnets,
      scalingConfig,
      enableDataApi,
      masterUsername,
    } = props;

    this.vpc = vpc;
    this.vpcSubnets = vpcSubnets;

    // DB subnet group
    const { subnetIds } = vpc.selectSubnets(vpcSubnets);

    const dbSubnetGroup = new rds.CfnDBSubnetGroup(this, 'DbSubnetGroup', {
      dbSubnetGroupDescription: `CloudFormation managed DB subnet group.`,
      subnetIds,
    });

    // DB security group
    const securityGroups = props.securityGroups ?? [
      new ec2.SecurityGroup(this, 'DbSecurityGroup', {
        vpc: vpc,
        allowAllOutbound: false,
      }),
    ];
    this.securityGroups = securityGroups;

    // DB secret
    const secret = new rds.DatabaseSecret(this, 'DbSecret', {
      username: masterUsername || 'root',
    });
    this.secretRotationApplication = engine.singleUserRotationApplication;

    // bind the engine to the Cluster
    const clusterEngineBindConfig = engine.bindToCluster(this, {
      parameterGroup: parameterGroup,
    });
    const clusterParameterGroup = parameterGroup ?? clusterEngineBindConfig.parameterGroup;
    const clusterParameterGroupConfig = clusterParameterGroup?.bindToCluster({});

    // DB cluster
    const cluster = new rds.CfnDBCluster(this, 'DbCluster', {
      // basic
      engine: engine.engineType,
      engineVersion: engine.engineVersion?.fullVersion,
      engineMode: 'serverless',
      dbClusterIdentifier: clusterIdentifier,
      dbSubnetGroupName: dbSubnetGroup.ref,
      vpcSecurityGroupIds: securityGroups.map((sg) => sg.securityGroupId),
      // port: clusterEngineBindConfig.port, // Aurora Serverless always runs on default port
      dbClusterParameterGroupName: clusterParameterGroupConfig?.parameterGroupName,
      // associatedRoles: undefined,
      // admin
      masterUsername: secret.secretValueFromJson('username').toString(),
      masterUserPassword: secret.secretValueFromJson('password').toString(),
      backupRetentionPeriod: backup?.retention.toDays(),
      preferredBackupWindow: backup?.preferredWindow,
      preferredMaintenanceWindow: preferredMaintenanceWindow,
      databaseName: defaultDatabaseName,
      deletionProtection: deletionProtection,
      // encryption
      kmsKeyId: storageEncryptionKey?.keyArn,
      // storageEncrypted: storageEncryptionKey ? true : encrypted, // Aurora Serverless is always encrypted
      // serverless config
      enableHttpEndpoint: enableDataApi,
      scalingConfiguration: scalingConfig,
    });

    // Default deletion policy for AWS:RDS:DBCluster is SNAPSHOT
    // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html
    if (removalPolicy) {
      cluster.applyRemovalPolicy(removalPolicy, {
        applyToUpdateReplacePolicy: true,
      });
    } else {
      // The CFN default doesn't cover UpdateReplacePolicy. Fix that here.
      cluster.cfnOptions.updateReplacePolicy = cdk.CfnDeletionPolicy.SNAPSHOT;
    }

    this.clusterIdentifier = cluster.ref;

    this.secret = secret.attach(this);

    this.clusterEndpoint = new rds.Endpoint(cluster.attrEndpointAddress, cdk.Token.asNumber(cluster.attrEndpointPort));

    const defaultPort = ec2.Port.tcp(this.clusterEndpoint.port);
    this.connections = new ec2.Connections({ securityGroups, defaultPort });

    const { region, account } = this.stack;
    this.clusterArn = `arn:aws:rds:${region}:${account}:cluster:${this.clusterIdentifier}`;
  }

  /**
   * Adds the single user rotation of the master password to this cluster.
   */
  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L542
  public addRotationSingleUser(automaticallyAfter?: cdk.Duration): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add single user rotation for a cluster without secret.');
    }

    const id = 'RotationSingleUser';
    const existing = this.node.tryFindChild(id);
    if (existing) {
      throw new Error('A single user rotation was already added to this cluster.');
    }

    return new secretsmanager.SecretRotation(this, id, {
      secret: this.secret,
      automaticallyAfter,
      application: this.secretRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }

  public asSecretAttachmentTarget(): secretsmanager.SecretAttachmentTargetProps {
    return {
      targetType: secretsmanager.AttachmentTargetType.RDS_DB_CLUSTER,
      targetId: this.clusterIdentifier,
    };
  }

  /*
  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L566
  public addRotationMultiUser(id: string, options: rds.RotationMultiUserOptions): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add multi user rotation for a cluster without secret.');
    }
    return new secretsmanager.SecretRotation(this, id, {
      secret: options.secret,
      masterSecret: this.secret,
      automaticallyAfter: options.automaticallyAfter,
      application: this.multiUserRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }
   */
}

I think creating a separate resource is probably the best option considering most of the options on the current cluster type don’t quite apply. E.g only MySQL and Postgres are available, and you can’t call lambdas from stored procedures or queries, etc.

Serverless also has the “Web API” feature that may be useful to turn on from some stacks. (Though, I’m not sure that it’s currently supported in Cfn)

You can use Escape Hatches to modify the L2 Aurora Server construct into a Serverless version:

from aws_cdk import (
    aws_ec2 as ec2,
    aws_rds as rds,
    core
)

class CdkRdsPipelineStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(
            self, "MyVpc",
            max_azs=2
        )

        aurora_sg = ec2.SecurityGroup(
            self, 'AU-SG',
            vpc=vpc,
            description="Allows MySQL connections",
        )

        # Aurora MySQL Cluster with Secrets Manager managed admin user
        aurora_cluster = rds.DatabaseCluster(
            self, "AU-CLUSTER-1",
            engine=rds.DatabaseClusterEngine.AURORA,
            engine_version="5.6.10a", # aws rds describe-db-engine-versions --engine aurora --query "DBEngineVersions[].EngineVersion" --region eu-west-1
            master_user=rds.Login(
                username='admin'
            ),
            default_database_name='MyDatabase',
            instance_props=rds.InstanceProps(
                instance_type=ec2.InstanceType('t3.small'),
                vpc=vpc,
                security_group=aurora_sg,
            ), 
            instances=1 # delete this 'Server' in an escape hatch below
        )

        # escape hatch https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html#cfn_layer_raw
        cfn_aurora_cluster = aurora_cluster.node.default_child
        cfn_aurora_cluster.add_override("Properties.EngineMode", "serverless")
        cfn_aurora_cluster.add_override("Properties.EnableHttpEndpoint",True) # Enable Data API
        cfn_aurora_cluster.add_override("Properties.ScalingConfiguration", { 
            'AutoPause': True, 
            'MaxCapacity': 4, 
            'MinCapacity': 1, 
            'SecondsUntilAutoPause': 600
        }) 
        aurora_cluster.node.try_remove_child('Instance1') # Remove 'Server' instance that isn't required for serverless Aurora

        # Add Secrets Manager Password rotation
        aurora_cluster.add_rotation_single_user()

This doc shows how to do property overrides.

Thanks for reporting. As a workaround, you should be able to use property overrides

I built on some of the work by @ApocDev and @StevenAskwith, in case it helps anyone else here:

  const vpc = new Vpc(this, 'Vpc', {maxAzs: 2});

  const securityGroup = new SecurityGroup(this, 'AuroraSecurityGroup', {
    vpc,
    allowAllOutbound: true,
  });

  const secret = new DatabaseSecret(this, 'AuroraSecret', {
    username: 'root',
  });

  const cluster = new CfnDBCluster(this, 'AuroraCluster', {
    engine: 'aurora',
    engineVersion: '5.6.10a',
    engineMode: 'serverless',
    masterUsername: secret.secretValueFromJson('username').toString(),
    masterUserPassword: secret.secretValueFromJson('password').toString(),
    vpcSecurityGroupIds: [securityGroup.securityGroupId],
    deletionProtection: false,
    enableHttpEndpoint: true,
    storageEncrypted: true,
    backupRetentionPeriod: 7,

    dbSubnetGroupName: new CfnDBSubnetGroup(this, 'AuroraSubnetGroup', {
      dbSubnetGroupDescription: 'AuroraSubnetGroup',
      subnetIds: vpc.selectSubnets({subnetType: SubnetType.PRIVATE}).subnetIds,
    }).ref,

    scalingConfiguration: {
      autoPause: true,
      minCapacity: 1,
      maxCapacity: 16,
      secondsUntilAutoPause: 300,
    },
  });

@skinny85

This still seems problematic (or at least not clearly documented).

For instance, engine.bindToCluster is required - but what is it? I don’t see it documented anywhere. The docs say it’s a method, but when I’m defining my cluster using rds.ServerlessCluster, it treats it as a required property.

Also, why are singleUserRotationApplication and multiUserRotationApplication needed? I’ve created a serverless cluster in CloudFormation YAML before and not needed these.

Lastly, it seems impossible to create a serverless cluster without NAT gateways, which are not required. If I use isolated subnets for the rds.SubnetGroup, it complains:

Error: There are no 'Private' subnet groups in this VPC. Available types: Isolated,Public

But if I create private subnets in my VPC definition, NAT gateways are automatically created. Setting natGateways: 0 complains that I should use isolated subnets—so you can see my catch-22 here.

  const secret = new rds.DatabaseSecret(stack, 'RDSSecret', {
    secretName: 'dev/rds',
    username: process.env.APP_NAME,
  });
  const subnetGroup = new rds.SubnetGroup(stack, 'RDSSubnetGroup', {
    subnetGroupName: 'rds-subnet-group',
    vpc: props.vpc,
    description: 'Private subnet group for DBs',
  });
  const db = new rds.ServerlessCluster(stack, 'Database', {
    backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 7),
    clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    credentials: {
      username: process.env.APP_NAME,
      secret,
    },
    subnetGroup,
    defaultDatabaseName: process.env.APP_NAME,
    deletionProtection: stack.stage === 'prod',
    enableDataApi: true,
    engine: {
      engineType: 'aurora-postgresql',
      engineVersion: {
        majorVersion: '10',
        fullVersion: '10.7',
      },
      // Why are these rotation values needed?
      multiUserRotationApplication:
        secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_MULTI_USER,
      singleUserRotationApplication:
        secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
      supportedLogTypes: ['what is this? not documented'],
      // What is this?
      bindToCluster({}, {}) {},
    },
    scaling: {
      autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
      minCapacity: rds.AuroraCapacityUnit.ACU_1,
      maxCapacity:
        stack.stage === 'prod'
          ? rds.AuroraCapacityUnit.ACU_8
          : rds.AuroraCapacityUnit.ACU_1,
    },
  });

I would use this

@skinny85 No sir! This wound up working fine for me:

  const secret = new rds.DatabaseSecret(stack, 'RDSSecret', {
    secretName: 'dev/rds',
    username: process.env.APP_NAME,
  });

  const mainVpc = ec2.Vpc.fromLookup(stack, 'MainVpc', { isDefault: true });

  const dbSecGroup = new ec2.SecurityGroup(stack, 'RDSSecurityGroup', {
    description: 'Access to DBs in VPC',
    securityGroupName: 'database-access',
    vpc: mainVpc,
    allowAllOutbound: true,
  });

  dbSecGroup.addIngressRule(dbSecGroup, ec2.Port.tcp(5432));

  const db = new rds.ServerlessCluster(stack, 'RDSDatabase', {
    backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 1),
    clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    credentials: rds.Credentials.fromSecret(secret),
    defaultDatabaseName: process.env.APP_NAME,
    enableDataApi: true,
    engine: rds.DatabaseClusterEngine.auroraPostgres({
      version: rds.AuroraPostgresEngineVersion.VER_10_14,
    }),
    scaling: {
      autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
      maxCapacity:
        stack.stage === 'prod'
          ? rds.AuroraCapacityUnit.ACU_8
          : rds.AuroraCapacityUnit.ACU_2,
      minCapacity: rds.AuroraCapacityUnit.ACU_2,
    },
    vpc: mainVpc,
    vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
    securityGroups: [dbSecGroup],
  });

I made some progress on this, but there are still some questionable things going on.

Here’s my definition of my serverless cluster using CfnDbCluster:

  const db = new rds.CfnDBCluster(stack, 'RDSDatabase', {
    backupRetentionPeriod: stack.stage === 'prod' ? 30 : 1,
    databaseName: process.env.APP_NAME,
    dbClusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    enableHttpEndpoint: true,
    engine: 'aurora-postgresql',
    engineMode: 'serverless',
    engineVersion: '10.14',
    masterUsername: secret.secretValueFromJson('username').toString(),
    masterUserPassword: secret.secretValueFromJson('password').toString(),
    vpcSecurityGroupIds: [dbSecGroup.securityGroupId],
    scalingConfiguration: {
      autoPause: stack.stage !== 'prod',
      maxCapacity: stack.stage === 'prod' ? 8 : 2,
      minCapacity: 2,
      secondsUntilAutoPause: 3600,
    },
  });

Here’s the ServerlessCluster version of that:

  const db = new rds.ServerlessCluster(stack, 'RDSDatabase', {
    backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 1),
    clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
    credentials: rds.Credentials.fromSecret(secret),
    defaultDatabaseName: process.env.APP_NAME,
    enableDataApi: true,
    engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
    scaling: {
      autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
      maxCapacity:
        stack.stage === 'prod'
          ? rds.AuroraCapacityUnit.ACU_8
          : rds.AuroraCapacityUnit.ACU_2,
      minCapacity: rds.AuroraCapacityUnit.ACU_2,
    },
    vpc: ec2.Vpc.fromLookup(stack, 'MainVpc', { isDefault: true }),
    securityGroups: [dbSecGroup],
  });

I get this error when trying to deploy: Error: There are no 'Private' subnet groups in this VPC. Available types: Public

Yet using CfnDbCluster, it works fine. Why are private subnets required when they’re really not? The security group is only so a dev can manually connect to the DB via SSH within SSM… otherwise it’s not needed since I’m using the Data API.

this article saved my day 🙇 🙇 🙇

I see, would it be preferable to add a “Serverless” Database Prop or instead change DatabaseClusterProps to support “Serverless” configurations?

Does #688 not implement this (it adds Aurora RDS)? (Side note, @eladb your link is no longer valid, have a new one?)

No. The DatabaseCluster doesn’t expose the engineMode and scalingConfiguration properties. It also has other property requirements that are only valid for non-serverless instances. (Instance type, size, etc) Those options are not available in the serverless mode.