aws-cdk: (aws-kinesis): fails to deploy on-demand stream

⚠️ Edit: Caution this might only work for creating a new stream from scratch (see comment below)

In the meantime, a quick fix is to manually set the shardCount prop to undefined on the Cfn resource:

const stream = new Stream(this, 'KinesisStream', {
    streamName: 'my-ingest-stream',
    streamMode: StreamMode.ON_DEMAND,
});

;(stream.node.defaultChild as CfnStream).shardCount = undefined

I have tested it and I am able to confirm that “ShardCount” is no longer present in the Cfn template.

Thanks @igilham for detailed workaround ! Upgrading to CDK version 1.138.2 fixed the issue.

@Harshit22 This issue is fixed in CDK versions 1.138.2 and 2.5.0.

The buggy version can be worked around using the code posted above by @pierre-vr.

Modifying it to match your example:

const dataArchivalKinesisStream = new Stream(this, 'KinesisStream', {
    streamName: `${props.stageName}-${props.realm}-DataArchivalKinesisStream`,
    retentionPeriod: Duration.days(14),
    streamMode: StreamMode.ON_DEMAND,
});
(dataArchivalKinesisStreamnode.defaultChild as CfnStream).shardCount = undefined

I have just created the following minimal test project to demonstrate that the fix works (using CDK 2.5.0):

app.ts

import { Construct } from 'constructs';
import { App, Duration, Stack, StackProps } from 'aws-cdk-lib';
import { Stream, StreamMode } from 'aws-cdk-lib/aws-kinesis';

const app = new App();

class MyStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    new Stream(this, 'stream', {
      streamName: 'my-stream',
      retentionPeriod: Duration.days(14),
      streamMode: StreamMode.ON_DEMAND
    });
  }
}

new MyStack(app, 'MyStack');

app.synth();

cdk.json

{
  "app": "npx ts-node --prefer-ts-exts lib/index.ts"
}

command to run

npx cdk synth

Output cdk.out/MyStack.template.json

{
  "Resources": {
    "stream19075594": {
      "Type": "AWS::Kinesis::Stream",
      "Properties": {
        "Name": "my-stream",
        "RetentionPeriodHours": 336,
        "StreamEncryption": {
          "Fn::If": [
            "AwsCdkKinesisEncryptedStreamsUnsupportedRegions",
            {
              "Ref": "AWS::NoValue"
            },
            {
              "EncryptionType": "KMS",
              "KeyId": "alias/aws/kinesis"
            }
          ]
        },
        "StreamModeDetails": {
          "StreamMode": "ON_DEMAND"
        }
      },
      "Metadata": {
        "aws:cdk:path": "MyStack/stream/Resource"
      }
    },
    "CDKMetadata": {
      "Type": "AWS::CDK::Metadata",
      "Properties": {
        "Analytics": "v2:deflate64:H4sIAAAAAAAA/yWIXQ5EMBCAz+K9HSziAG7AAaTbjmSUadIpHjbuvhpP388HOqgKc4m2zuuNvvCbkrFePWv2xCgkeUU0uxoWfu3OOgR2lChwjhElHNHirTg4hFXKs26h7qEpViHS8eBEO8L48g+nSx5idwAAAA=="
      },
      "Metadata": {
        "aws:cdk:path": "MyStack/CDKMetadata/Default"
      },
      "Condition": "CDKMetadataAvailable"
    }
  },
  "Conditions": {
    "AwsCdkKinesisEncryptedStreamsUnsupportedRegions": {
      "Fn::Or": [
        {
          "Fn::Equals": [
            {
              "Ref": "AWS::Region"
            },
            "cn-north-1"
          ]
        },
        {
          "Fn::Equals": [
            {
              "Ref": "AWS::Region"
            },
            "cn-northwest-1"
          ]
        }
      ]
    },
    "CDKMetadataAvailable": {
      "Fn::Or": [
        {
          "Fn::Or": [
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "af-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-east-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-northeast-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-northeast-2"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-southeast-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-southeast-2"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ca-central-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "cn-north-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "cn-northwest-1"
              ]
            }
          ]
        },
        {
          "Fn::Or": [
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-central-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-north-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-west-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-west-2"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-west-3"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "me-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "sa-east-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-east-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-east-2"
              ]
            }
          ]
        },
        {
          "Fn::Or": [
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-west-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-west-2"
              ]
            }
          ]
        }
      ]
    }
  },
  "Parameters": {
    "BootstrapVersion": {
      "Type": "AWS::SSM::Parameter::Value<String>",
      "Default": "/cdk-bootstrap/hnb659fds/version",
      "Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
    }
  },
  "Rules": {
    "CheckBootstrapVersion": {
      "Assertions": [
        {
          "Assert": {
            "Fn::Not": [
              {
                "Fn::Contains": [
                  [
                    "1",
                    "2",
                    "3",
                    "4",
                    "5"
                  ],
                  {
                    "Ref": "BootstrapVersion"
                  }
                ]
              }
            ]
          },
          "AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
        }
      ]
    }
  }
}

I have confirmed @pierre-vr’s work-around is successful for streams that have been manually marked as on-demand and therefore drifted from the deployed stack template, which had a specific shardCount.

@d4r3topk I’m working through these scenarios now.

I initially thought it would be easiest to remove default values from the generated CloudFormation and allow the upstream service’s defaults to take over. Now that I’ve been spending a bit more time with CDK’s codebase, this seems not to be the preferred pattern, though. CDK likes to be explicit where possible.

I’m now following the example set by Table in the DynamoDB module. This uses guard statements in the L2 constructor to ensure the correct defaults are applied in each scenario, or an error is thrown if the user sets incompatible options.

I can’t get the tests to run on my computer so I’m relying on the CodeBuild infrastructure connected to the Pull Request. I hope to finish getting it ready today.

I’d be happy to fix the bug when I have time. It’ll probably be next week. I’m not familiar with all the testing layers so doing a thorough job may not be such a quick fix but I’m happy to take a look.

If anyone wants to do it sooner than I can, I won’t be offended but I’ll aim to pick this up soon if nobody else has.

Since I wrote the code for this feature, I can only apologise for not testing it through CloudFormation before shipping it.