aws-cdk: [core] unable to run CDK diff in upgrading from 1.51.0 to 1.62.0: : Assets must be defined indirectly within a "Stage" or an "App" scope

I just upgraded my cdk version from 1.51.0 to 1.62.0, and I updated the code so that npm build runs successfully for the project. I then tried to do a diff of my stacks with cdk diff <stackname> and it failed with this error:

unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1

In googling that I came across https://github.com/aws/aws-cdk/issues/9546 and I followed the directions to make sure all my cdk packages were pinned at the exact same version. after doing that I still ran into the error. I do use custom lambda functions packaged like so:

 const statusLambda = new lambda.Function(this, 'StatusLambda', {
            code: lambda.Code.asset(path.join(__dirname, '../lambdas/codepipeline_status_lambda')),
            handler: 'index.handler',
            timeout: cdk.Duration.seconds(300),
            runtime: lambda.Runtime.NODEJS_10_X,
            environment: {
                ACCESS_TOKEN: cdk.SecretValue.secretsManager(props.gitHubSecretArn).toString(),
                DATADOG_API_KEY: cdk.SecretValue.secretsManager(props.dataDogSecretArn).toString(),
                ENV_TYPE: props.env_type,
                STAGE_TYPE: props.stage_type,
                APP_URL: props.url,
            },
        })

which worked just fine before the upgrade, but now it still is failing with the same error, that issue i linked above mentioned that using lambda assets could cause this. and I am unsure of how to proceed to fix this, and also why this suddenly became an issue.

Reproduction Steps

use a custom lambda asset like so:

 const statusLambda = new lambda.Function(this, 'StatusLambda', {
            code: lambda.Code.asset(path.join(__dirname, '../lambdas/codepipeline_status_lambda')),
            handler: 'index.handler',
            timeout: cdk.Duration.seconds(300),
            runtime: lambda.Runtime.NODEJS_10_X,
            environment: {
                ACCESS_TOKEN: cdk.SecretValue.secretsManager(props.gitHubSecretArn).toString(),
                DATADOG_API_KEY: cdk.SecretValue.secretsManager(props.dataDogSecretArn).toString(),
                ENV_TYPE: props.env_type,
                STAGE_TYPE: props.stage_type,
                APP_URL: props.url,
            },
        })

What did you expect to happen?

cdk diff to pass without error

What actually happened?

cdk diff returned this error:

unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1

Verbose output:

CDK toolkit version: 1.62.0 (build 8c2d7fc)
Command line arguments: {
  _: [ 'diff' ],
  profile: 'outline-dev',
  v: 4,
  verbose: 4,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  fail: false,
  'context-lines': 3,
  contextLines: 3,
  strict: false,
  '$0': 'cdk',
  STACKS: [ 'DevCoreStack' ],
  stacks: [ 'DevCoreStack' ]
}
cdk.json: {
  "app": "npx ts-node bin/corestack.ts"
}
cdk.context.json: {
  "availability-zones:account=349142687622:region=us-west-2": [
    "us-west-2a",
    "us-west-2b",
    "us-west-2c",
    "us-west-2d"
  ],
  "availability-zones:account=349142687622:region=us-east-1": [
    "us-east-1a",
    "us-east-1b",
    "us-east-1c",
    "us-east-1d",
    "us-east-1e",
    "us-east-1f"
  ],
  "availability-zones:account=703994937577:region=us-east-1": [
    "us-east-1a",
    "us-east-1b",
    "us-east-1c",
    "us-east-1d",
    "us-east-1e",
    "us-east-1f"
  ],
  "availability-zones:account=703994937577:region=us-east-2": [
    "us-east-2a",
    "us-east-2b",
    "us-east-2c"
  ]
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'npx ts-node bin/corestack.ts',
  context: {},
  assetMetadata: true,
  profile: 'outline-dev',
  toolkitBucket: {},
  staging: true
}
Determining if we're on an EC2 instance.
Does not look like an EC2 instance.
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to us-east-1
Resolving default credentials
Retrieved account ID 349142687622 from disk cache
Setting "CDK_DEFAULT_ACCOUNT" environment variable to 349142687622
context: {
  'availability-zones:account=349142687622:region=us-west-2': [ 'us-west-2a', 'us-west-2b', 'us-west-2c', 'us-west-2d' ],
  'availability-zones:account=349142687622:region=us-east-1': [
    'us-east-1a',
    'us-east-1b',
    'us-east-1c',
    'us-east-1d',
    'us-east-1e',
    'us-east-1f'
  ],
  'availability-zones:account=703994937577:region=us-east-1': [
    'us-east-1a',
    'us-east-1b',
    'us-east-1c',
    'us-east-1d',
    'us-east-1e',
    'us-east-1f'
  ],
  'availability-zones:account=703994937577:region=us-east-2': [ 'us-east-2a', 'us-east-2b', 'us-east-2c' ],
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true
}
outdir: cdk.out
env: {
  CDK_DEFAULT_REGION: 'us-east-1',
  CDK_DEFAULT_ACCOUNT: '349142687622',
  CDK_CONTEXT_JSON: '{"availability-zones:account=349142687622:region=us-west-2":["us-west-2a","us-west-2b","us-west-2c","us-west-2d"],"availability-zones:account=349142687622:region=us-east-1":["us-east-1a","us-east-1b","us-east-1c","us-east-1d","us-east-1e","us-east-1f"],"availability-zones:account=703994937577:region=us-east-1":["us-east-1a","us-east-1b","us-east-1c","us-east-1d","us-east-1e","us-east-1f"],"availability-zones:account=703994937577:region=us-east-2":["us-east-2a","us-east-2b","us-east-2c"],"aws:cdk:enable-path-metadata":true,"aws:cdk:enable-asset-metadata":true}',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '5.0.0',
  CDK_CLI_VERSION: '1.62.0'
}
unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1
Error: Subprocess exited with error 1
    at ChildProcess.<anonymous> (/Users/grahambooth/.nvm/versions/node/v12.18.3/lib/node_modules/aws-cdk/lib/api/cxapp/exec.ts:118:23)
    at ChildProcess.emit (events.js:315:20)
    at ChildProcess.EventEmitter.emit (domain.js:483:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)

Environment

  • CLI Version : 1.62.0
  • Framework Version:
  • Node.js Version:v12.18.3
  • OS : macOS
  • Language (Version):TypeScript (3.8.3)

Other


This is šŸ› Bug Report

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 21 (9 by maintainers)

Commits related to this issue

Most upvoted comments

I think I’ve tracked this issue down.

As @jogold pointed out, it has to do with nesting different copies of @aws-cdk/core. However I politely disagree and would argue it is a very valid use case, especially with monorepos. It occurs whenever packages aren’t hoisted to the top. Lerna provides a mode for this and even npm-v7 workspaces seem to encourage this node_modules structure.

Generally speaking I don’t think it sits well with CDK to enforce a single instance of the code, not just the same version number.

Therefore the example repo @Shogan created is still a good starting point. The issue occurs with any kind of assets. Lambda Functions with Code Assets are an obvious one as they are often pulled out into a reusable library. However like @flochaz this also appears to happen when the code asset is just a parameter to a lambda lib. The reason being that a lambda with logRetention will cause assets for the custom resource to be stages, which will then fail.


Enough of the prelude. The issue surfaces because stages are enforced for asset staging as of 1.57.0. Specifically this change by @eladb

Now, Stages have been around for a long time. However they have been pretty much optional and I guess where only used with the new pipelines. Long story short, they use a different mechanic to determine if a thing is a stage object.

The Stage class uses instanceof which will return false for objects that are not from the same code instance.

All other ā€œglobalā€ packages use a mechanism with Symbol.for: App, Stack, Aspect


Unless there is a good reason for this difference, I’d suggest to use the same mechanism for Stages which will fix this issue. If someone could confirm the direction, I’m happy to provide a PR as it is holding us back.

I’d also suggest to include a test for nested code copies.

We’ve just updated a bunch of our lerna stacks to 1.72.0 and it finally works again! šŸŽ‰

I am hitting this bug with CDK versions 1.64.0 and 1.64.1.

@Shogan Can you share a repro?

I’ve created the simplest example I can think of, reproducing the problem here: https://github.com/gwriss/upgrade_bug

Extra useful information about the issue can be found here: https://github.com/aws/aws-cdk/issues/10977

Please let me know if there is anything I can do to help solve the issue

@Shogan you are nesting different copies of @aws-cdk/core (same version but different copies) this is the problem, see https://github.com/aws/aws-cdk/issues/10210#issuecomment-690093916 for an explanation.

You should use a monorepo/lerna for this kind of architecture.

I am hitting this bug with CDK versions 1.64.0 and 1.64.1.

@shogan Can you share a repro?

I am hitting this bug with CDK versions 1.64.0 and 1.64.1. There are two ways of triggering this bug:

  1. Mismatched versions of cdk modules in a typescript project
  2. Creating a Lambda function using source code that is nested in a different folder e.g. up to parent directory and then within a sub-directory of that, or in a child directory of the current working directory.

I am experiencing number 2 above (lambda with Code.fromAsset).

Here is an example:

let lambdaFunc = new lambda.Function(this, `MyFunc`, {
      runtime: lambda.Runtime.NODEJS_12_X,
      code: lambda.Code.fromAsset(path.join(__dirname, 'lambda/myfunc')),
      handler: "index.handler",
      memorySize: 128,
      timeout: Duration.seconds(5)
});

I’ve tried deleting node_modules, package-lock.json, and cdk.out, and have triple checked all aws-cdk packages are at the exact same, pinned version. Then done npm install && tsc. cdk diff or cdk synth or cdk deploy all result in the error:

unable to determine cloud assembly output directory. Assets must be defined indirectly within a "Stage" or an "App" scope
Subprocess exited with error 1

As soon as I downgrade to 1.52.0 (or 1.57.0) then everything works fine.