aws-cdk: core: Custom::CrossRegionExportWriter fails with InvalidResourceId: UnknownError

Describe the bug

I’m using crossRegionReferences to use my Hosted Zone created in region eu-west-1 for my Certificate created in us-east-1 but the exported value is never created and fails with InvalidResourceId: UnknownError when I deploy. Downgrading to 2.69 fixes the issue.

Expected Behavior

The deploy should work without any errors. The SSM parameter should be created.

Current Behavior

The deploy fails with the following error:

Error processing event:  InvalidResourceId: UnknownError
    at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
    at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
    at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at async isInUse (/var/task/index.js:5:87)
    at async /var/task/index.js:3:932
    at async Promise.all (index 0) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: 'ff52df1c-c7bd-44cd-8c70-9ef10075ef40',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'InvalidResourceId'
}

CREATE failed, responding with a marker physical resource id so that the subsequent DELETE will be ignored

Request:

{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:eu-west-1:***:function:HostedZoneStack-CustomCrossRegionExportWriterCusto-oj2kKqRu7LRG",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:eu-west-1:***:stack/HostedZoneStack/c980a370-20be-11ee-b44d-0a884cc475df",
    "RequestId": "ae7984ed-4a67-4ed4-b3df-6c60d1d23889",
    "LogicalResourceId": "ExportsWriteruseast10F67B507DDE2E818",
    "ResourceType": "Custom::CrossRegionExportWriter",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:eu-west-1:***:function:HostedZoneStack-CustomCrossRegionExportWriterCusto-oj2kKqRu7LRG",
        "WriterProps": {
            "exports": {
                "/cdk/exports/CertificateWAF2Stack/HostedZoneStackeuwest1RefHoztedZoneStackHostedZone662869C91D9E1585": "Z0515350ARL1Q47HQ75Z"
            },
            "region": "us-east-1"
        }
    }
}

Reproduction Steps

const hostedZoneStack = new HoztedZoneStack(app, 'HostedZoneStack', {
    zoneName: AWS_DOMAIN,
    env: AWS_ENV,
    crossRegionReferences: true
});

const certificateStack = new CertificateWaf2Stack(app, 'CertificateWAF2Stack', {
    domainName: AWS_DOMAIN,
    hostedZone: hostedZoneStack.hostedZone,
    aclName: ACL_NAME,
    wafScope: WAF_SCOPE,
    env: AWS_ENV_GLOBAL,
    crossRegionReferences: true
});

import { Stack, StackProps, Tags } from 'aws-cdk-lib';
import { HostedZone } from 'aws-cdk-lib/aws-route53';
import { Construct } from 'constructs';

type Props = StackProps & {
    zoneName: string;
};

export class HoztedZoneStack extends Stack {
    public readonly hostedZone: HostedZone;

    constructor(scope: Construct, id: string, props: Props) {
        super(scope, id, props);

        this.hostedZone = new HostedZone(this, 'HoztedZoneStackHostedZone', {
            zoneName: props.zoneName
        });

        // Add tags
        Tags.of(this.hostedZone).add('Name', props.zoneName);
    }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0

Framework Version

No response

Node.js Version

18.14.1

OS

Windows 10 x64

Language

Typescript

Language Version

5.1.6

Other information

No response

About this issue

Original URL
State: closed
Created a year ago
Reactions: 7
Comments: 15 (7 by maintainers)

Most upvoted comments

Making this a p1 bug. The maintainer is investigating now.

pahud on Jul 13, 2023

This may have been introduced in 2.87.0, downgrading to 2.86.0 fixed the issue for me.

thomasedw on Jul 16, 2023

Yes I can reproduce this issue even with the basic cross-region reference.

#!/usr/bin/env node
import 'source-map-support/register';
import {
App, Stack, StackProps, CfnOutput,
aws_ec2 as ec2,
} from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class Stack2 extends Stack {
  public readonly vpc: ec2.IVpc
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    this.vpc = new ec2.Vpc(this,'Vpc',{
      natGateways: 1
    });
  }
}

export interface stack3props extends StackProps {
  readonly vpc: ec2.IVpc;
  }

export class Stack3 extends Stack {
  
  constructor(scope: Construct, id: string, props: stack3props) {
    super(scope, id, props);

    new CfnOutput(this, 'VpcId', { value: props.vpc.vpcId})
}}

const app = new App();
// const env = { region: process.env.CDK_DEFAULT_REGION, account: process.env.CDK_DEFAULT_ACCOUNT };

const envUSA = { account: process.env.CDK_DEFAULT_ACCOUNT , region: 'us-east-2' };
const envUSA1 = { account: process.env.CDK_DEFAULT_ACCOUNT , region: 'eu-central-1' };

const stack2 = new Stack2(app,'teststack2',{
    env: envUSA,
    crossRegionReferences: true
});
  
new Stack3(app,'teststack3',{
    env: envUSA1,
    crossRegionReferences: true,
    vpc: stack2.vpc
});

error

teststack2: creating CloudFormation changeset...
4:14:00 PM | CREATE_FAILED        | Custom::CrossRegionExportWriter       | ExportsWritereucentral1E172851B74269898
Received response status [FAILED] from custom resource. Message returned: InvalidResourceId: UnknownError
at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25
)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
at async isInUse (/var/task/index.js:5:87)
at async /var/task/index.js:3:932
at async Promise.all (index 0) (RequestId: f7945e26-4e08-4345-9608-2ab1891f3b37)


 ❌  teststack2 failed: Error: The stack named teststack2 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: InvalidResourceId: UnknownError
    at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
    at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /var/runtime/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:20
    at async StandardRetryStrategy.retry (/var/runtime/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at async isInUse (/var/task/index.js:5:87)
    at async /var/task/index.js:3:932
    at async Promise.all (index 0) (RequestId: f7945e26-4e08-4345-9608-2ab1891f3b37)
    at FullCloudFormationDeployment.monitorDeployment (/projects/triage/node_modules/aws-cdk/lib/index.js:412:10236)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.deployStack2 [as deployStack] (/projects/triage/node_modules/aws-cdk/lib/index.js:415:153172)
    at async /projects/triage/node_modules/aws-cdk/lib/index.js:415:136968

 ❌ Deployment failed: Error: The stack named teststack2 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: InvalidResourceId: UnknownError
    at deserializeAws_json1_1InvalidResourceIdResponse (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:7669:23)
    at deserializeAws_json1_1ListTagsForResourceCommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/protocols/Aws_json1_1.js:5473:25)

pahud on Jul 14, 2023

Closed by https://github.com/aws/aws-cdk/pull/26434

mrgrain on Jul 20, 2023