aws-cdk: ❗NOTICE: Cross-account deployments are failing in v2.163.0 with error 'Need to perform AWS calls for account'

Please add your +1 👍 to let us know you have encountered this


Status: RESOLVED

Overview:

Release v2.163.0 introduced an issues with cross-account deployments. The AWS CDK fails with the error Need to perform AWS calls for account XXXXX, but the current credentials are for YYYYY. The root cause is a new check for the bootstrap version added in v2.163.0. The check needs to make an API call to AWS CloudFormation to describe the bootstrap stack. This API call was using incorrect credentials.

Complete Error Message:

Need to perform AWS calls for account XXXXXXXXXXXX, but the current credentials are for YYYYYYYYYYYY

Workaround:

Downgrade the AWS CDK CLI to version 2.162.1

Solution:

Upgrade the AWS CDK CLI to version 2.163.1

Related Issues:

n/a


Original issue:

Describe the bug

Release v2.163.0 causes deployments using cross account role assumption to fail with the error Need to perform *** calls for account XXXXX, but the current credentials are for YYYYY. This is presumably caused by #31623. Based on comments on this PR (as it has no description or linked issue) this was done to address a vulnerability when the assets bucket for an account is not longer in the target account, but this failure is occurring when the assets bucket is still in the target account with no change. These deployments and buckets are in the ap-southeast-2 region.

Regression Issue

Confirmed Regression

Last Known Working CDK Version

2.162.1

Expected Behavior

Cross account deployments should succeed unless the asset bucket is no longer in the target account.

Current Behavior

Cross account deployments fail with the error Need to perform *** calls for account XXXXX, but the current credentials are for YYYYY cdk diff succeeds as expected, and cloudtrail shows role assumption is successful. Debug log excerpt just before the failure:

[04:15:56] datadog-lambda-apm-test-dev: check: Check s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2/2819175352ad1ce0dae768e83fc328fb70fb5f10b4a8ff0ccbcb791f02b0716d.zip
[04:15:56] [*** s3 200 0.06s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2',
  Prefix: 'fdfffca2098cb4f12a1dd3f0f2f98b4606063436cdfc311da0589251fa61cbe0.json',
  MaxKeys: 1
})
[04:15:56] [*** s3 200 0.074s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2',
  Prefix: '7fbaff7dbc84f82c58c19512abc1c6e165bebea0038d20da2d2ee153808fee70.zip',
  MaxKeys: 1
})
[04:15:56] datadog-lambda-apm-test-dev: found: Found s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2/7fbaff7dbc84f82c58c19512abc1c6e165bebea0038d20da2d2ee153808fee70.zip
[04:15:56] [*** s3 200 0.06s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2',
  Prefix: '2819175352ad1ce0dae768e83fc328fb70fb5f10b4a8ff0ccbcb791f02b0716d.zip',
  MaxKeys: 1
})
[04:15:56] datadog-lambda-apm-test-dev: found: Found s3://cdk-hnb659fds-assets-XXXXXXXXXXXX-ap-southeast-2/2819175352ad1ce0dae768e83fc328fb70fb5f10b4a8ff0ccbcb791f02b0716d.zip
[04:15:56] 3 total assets, 1 still need to be published
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#baseCredentialsPartition()
[04:15:56] [trace]   SdkProvider#resolveEnvironment()
[04:15:56] [trace]   SdkProvider#obtainBaseCredentials()
[04:15:56] [trace]     SdkProvider#defaultAccount()
[04:15:56] [trace]     SdkProvider#defaultCredentials()
[04:15:56] [trace]   SDK#currentAccount()
[04:15:56] [trace]     SDK#forceCredentialRetrieval()
[04:15:56] Retrieved account ID YYYYYYYYYYYY from disk cache
[04:15:56] [trace] SDK#ssm()
[04:15:56] [trace]   SDK#wrapServiceErrorHandling()
[04:15:56] [*** ssm 200 0.062s 0 retries] getParameter({ Name: '/cdk-bootstrap/hnb659fds/version' })
datadog-lambda-apm-test-dev: start: Building fdfffca2098cb4f12a1dd3f0f2f98b4606063436cdfc311da0589251fa61cbe0:XXXXXXXXXXXX-ap-southeast-2
datadog-lambda-apm-test-dev: success: Built fdfffca2098cb4f12a1dd3f0f2f98b4606063436cdfc311da0589251fa61cbe0:XXXXXXXXXXXX-ap-southeast-2
[04:15:56] [trace] SdkProvider#resolveEnvironment()
[04:15:56] [trace] SdkProvider#baseCredentialsPartition()
[04:15:56] [trace]   SdkProvider#resolveEnvironment()
[04:15:56] [trace]   SdkProvider#obtainBaseCredentials()
[04:15:56] [trace]     SdkProvider#defaultAccount()
[04:15:56] [trace]     SdkProvider#defaultCredentials()
[04:15:56] [trace]   SDK#currentAccount()
[04:15:56] [trace]     SDK#forceCredentialRetrieval()
[04:15:56] Retrieved account ID YYYYYYYYYYYY from disk cache
[04:15:56] [trace] SdkProvider#forEnvironment()
[04:15:56] [trace]   SdkProvider#resolveEnvironment()
[04:15:56] [trace]   SdkProvider#obtainBaseCredentials()
[04:15:56] [trace]     SdkProvider#defaultAccount()
[04:15:56] [trace]     SdkProvider#defaultCredentials()
[04:15:56] Notices refreshed
Need to perform *** calls for account XXXXXXXXXXXX, but the current credentials are for YYYYYYYYYYYY
[04:15:56] Error: Need to perform *** calls for account XXXXXXXXXXXX, but the current credentials are for YYYYYYYYYYYY
    at SdkProvider.forEnvironment (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/aws-auth/sdk-provider.ts:195:60)
    at Deployments.cachedSdkForEnvironment (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/deployments.ts:918:17)
    at Deployments.allowCrossAccountAssetPublishingForEnv (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/deployments.ts:863:20)
    at Deployments.publishSingleAsset (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/api/deployments.ts:855:62)
    at Object.publishAsset (/home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/cdk-toolkit.ts:254:7)
    at /home/runner/.npm/_npx/39b4457c6eead837/node_modules/aws-cdk/lib/util/work-graph.ts:111:11

Reproduction Steps

  1. Create a role in account A
  2. Bootstrap account B with account A (or the role in account A) as the trusted principle
  3. Assume role in account A
  4. Run npx cdk deploy on a stack with account B as the env.account property. May need to also set region to ap-southeast-2, I have not been able to test this in us-east-1 yet.

Possible Solution

Roll back #31623 until full integration testing can be performed.

Additional Information/Context

This change has caused all deployments from our CICD to fail, as they make use of cross account role assumption and use the latest CDK V2 CLI unless otherwise specified (due to caret version matching to avoid breaking changes). I would like to request that this change be rolled back until it can be thoroughly integration tested. If this is not possible, the new expected requirements for cross account CDK deployments need to be clearly documented.

CDK CLI Version

2.163.0

Framework Version

No response

Node.js Version

v20.13.1

OS

Debian Linux

Language

TypeScript

Language Version

TypeScript (5.6.3)

Other information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 5
  • Comments: 17 (7 by maintainers)

Commits related to this issue

Most upvoted comments

After further investigation, I can confirm this was caused by #31623, specifically this line: https://github.com/aws/aws-cdk/pull/31623/files#diff-1c33e80dabddc5697544f07e314339fa3abe37c68183cb882ce5127b76f7deeeR863

const sdk = (await this.cachedSdkForEnvironment(env, Mode.ForReading)).sdk;

All other uses of cachedSdkForEnvironment in this file also specify the assume role details:

const stackSdk = await this.cachedSdkForEnvironment(resolvedEnvironment, Mode.ForReading, {
  assumeRoleArn: arns.lookupRoleArn,
  assumeRoleExternalId: stack.lookupRole?.assumeRoleExternalId,
  assumeRoleAdditionalOptions: stack.lookupRole?.assumeRoleAdditionalOptions,
});

In a cross-account deployment allowCrossAccountAssetPublishingForEnv is now requesting SDK credentials for an env that doesnt match the base credentials, but not providing an assume role ARN. This hits the exception handling here throwing the exception seen in the logs.

Same issue here. Downgrading to cdk@2.162.1 resolves it for now.

v2.163.1 was released with a fix.

Can you please confirm if this resolves the issue for you?

I can confirm that 2.163.1 resolved the issue for us, thanks for the quick turnaround.

@rix0rrr - that was the issue, excellent catch. @mrgrain - I appreciate your time. Many thanks to you both, this is a big relief!