aws-cdk: integ-runner: Integration tests succeed, yet the integ-runner throws errors on "describeStackEvents" calls

Describe the bug

We recently upgraded from 2.93.0 to 2.100.0 … once we did, we saw a strange behavior with one of our integration tests on a large stack with nested stacks. The launching of the test stacks work fine (there are no errors in the CREATE or DELETE process), but during the tests we get errors printed out to the console that eventually cause the test-case itself to fail:

The test creation phase …

$ yarn integ test/integ/constructs/aws-eks/integ.xx-cluster.ts -vvvvv --parallel-regions=us-east-2
yarn run v1.22.19
warning package.json: "dependencies" has dependency "projen" with range "^0.73.8" that collides with a dependency in "devDependencies" of the same name with version "^0.73.35"
...
...
  test/integ/constructs/aws-eks/cdk-integ.out.integ.xx-cluster.ts.snapshot/bundling-temp-8d8dd6365075d75b5065745e14ac81d8b26551cbab856be8fee4b434bd30d711/index.js  1.3mb ⚠️


✨  Synthesis time: 5.73s

...

INFRA-CDK-K8S-ClusterTest: deploying... [1/1]
[20:01:22] Retrieved account ID 12345678910 from disk cache
[20:01:22] Call failed: describeStacks({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack with id INFRA-CDK-K8S-ClusterTest does not exist (code=ValidationError)
[20:01:22] INFRA-CDK-K8S-ClusterTest: checking if we can skip deploy
[20:01:22] INFRA-CDK-K8S-ClusterTest: no existing stack
[20:01:22] INFRA-CDK-K8S-ClusterTest: deploying...
[20:01:22] Attempting to create ChangeSet with name cdk-deploy-change-set to create stack INFRA-CDK-K8S-ClusterTest
INFRA-CDK-K8S-ClusterTest: creating CloudFormation changeset...
[20:01:23] Initiated creation of changeset: arn:aws:cloudformation:us-east-2:12345678910:changeSet/cdk-deploy-change-set/a72ea0de-91e5-4177-bece-d293fa36bb68; waiting for it to finish creating...
[20:01:23] Waiting for changeset cdk-deploy-change-set on stack INFRA-CDK-K8S-ClusterTest to finish creating...
[20:01:23] Changeset cdk-deploy-change-set on stack INFRA-CDK-K8S-ClusterTest is still creating
[20:01:29] Changeset cdk-deploy-change-set on stack INFRA-CDK-K8S-ClusterTest is still creating
[20:01:34] Initiating execution of changeset arn:aws:cloudformation:us-east-2:12345678910:changeSet/cdk-deploy-change-set/a72ea0de-91e5-4177-bece-d293fa36bb68 on stack INFRA-CDK-K8S-ClusterTest
[20:01:35] Execution of changeset arn:aws:cloudformation:us-east-2:12345678910:changeSet/cdk-deploy-change-set/a72ea0de-91e5-4177-bece-d293fa36bb68 on stack INFRA-CDK-K8S-ClusterTest has started; waiting for the update to complete...
[20:01:35] Waiting for stack INFRA-CDK-K8S-ClusterTest to finish creating or updating...
[20:01:35] Stack INFRA-CDK-K8S-ClusterTest has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS (User Initiated))
[20:01:40] Call failed: describeStackEvents({"StackName":""}) => 2 validation errors detected: Value '' at 'stackName' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]* (code=ValidationError)
Error occurred while monitoring stack: Error [ValidationError]: 2 validation errors detected: Value '' at 'stackName' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*
    at Request.extractError (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ValidationError',
  time: 2023-10-08T03:01:40.976Z,
  requestId: 'f65badfa-1f8e-405f-8e79-4790933ee422',
  statusCode: 400,
  retryable: false, 
  retryDelay: 568.6748870259721
} 
...
INFRA-CDK-K8S-ClusterTest-CorePluginsNestedStackCorePluginsNestedStackResource0210684D-SM0BQU1FZPFC | 26/32 | 8:17:25 PM | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | CorePlugins/AWSCDKCfnUtilsProviderCustomResourceProvider/Role (AWSCDKCfnUtilsProviderCustomResourceProviderRoleFE0EE867) Resource creation Initiated
[20:17:28] Call failed: describeStackEvents({"StackName":""}) => 2 validation errors detected: Value '' at 'stackName' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]* (code=ValidationError)
Error occurred while monitoring stack: Error [ValidationError]: 2 validation errors detected: Value '' at 'stackName' failed to satisfy constraint: Member must have length greater than or equal to 1; Value '' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*
    at Request.extractError (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'ValidationError',
  time: 2023-10-08T03:17:28.005Z,
  requestId: 'fe6608aa-c968-4ada-9199-5180c52b2983',
  statusCode: 400,
  retryable: false,
  retryDelay: 514.9915468965557
}
...
[20:25:25] Stack INFRA-CDK-K8S-ClusterTest has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
INFRA-CDK-K8S-ClusterTest | 28/32 | 8:25:29 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack            | SupplementalPlugins.NestedStack/SupplementalPlugins.NestedStackResource (SupplementalPluginsNestedStackSupplementalPluginsNestedStackResource924A2FB6)
[20:25:31] Stack INFRA-CDK-K8S-ClusterTest has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
INFRA-CDK-K8S-ClusterTest | 29/32 | 8:25:31 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack            | INFRA-CDK-K8S-ClusterTest
[20:25:36] Stack INFRA-CDK-K8S-ClusterTest has completed updating

 ✅  INFRA-CDK-K8S-ClusterTest

✨  Deployment time: 1454.39s

Outputs:
INFRA-CDK-K8S-ClusterTest.ClusterClusterNameEB26049E = xxx-integrationtest
Stack ARN:

✨  Total time: 1460.11s
...

Then there’s the deletion phase:

[20:25:36] Notices refreshed

[20:25:37] CDK toolkit version: 2.100.0 (build e1b5c77)
[20:25:37] Command line arguments: {
  _: [ 'deploy' ],
  rollback: false,
  'outputs-file': 'cdk-integ.out.integ.xx-cluster.ts.snapshot/assertion-results.json',
  O: 'cdk-integ.out.integ.xx-cluster.ts.snapshot/assertion-results.json',
  outputsFile: 'cdk-integ.out.integ.xx-cluster.ts.snapshot/assertion-results.json',
  'require-approval': 'never',
...
  test/integ/constructs/aws-eks/cdk-integ.out.integ.xx-cluster.ts.snapshot/bundling-temp-8d8dd6365075d75b5065745e14ac81d8b26551cbab856be8fee4b434bd30d711/index.js  1.3mb ⚠️


✨  Synthesis time: 37.49s

[20:26:14] Checking for previously published assets
,...
IntegTest/DefaultTest/DeployAssert: destroying... [1/2]
[20:26:22] Retrieved account ID 12345678910 from disk cache
[20:26:22] Assuming role 'arn:aws:iam::12345678910:role/cdk-hnb659fds-deploy-role-12345678910-us-east-2'.
[20:26:22] Call failed: describeStacks({"StackName":"IntegTestDefaultTestDeployAssertE3E7D2A4"}) => Stack with id IntegTestDefaultTestDeployAssertE3E7D2A4 does not exist (code=ValidationError)

 ✅  IntegTest/DefaultTest/DeployAssert: destroyed
INFRA-CDK-K8S-ClusterTest: destroying... [2/2]
[20:26:22] Retrieved account ID 12345678910 from disk cache
[20:26:23] Waiting for stack INFRA-CDK-K8S-ClusterTest to finish creating or updating...
[20:26:23] Stack INFRA-CDK-K8S-ClusterTest has an ongoing operation in progress and is not stable (DELETE_IN_PROGRESS (User Initiated))
INFRA-CDK-K8S-ClusterTest-SupplementalPluginsNestedStackSupplementalPluginsNestedStack-1Y25214IX1XHB |   0 | 8:26:25 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack            | INFRA-CDK-K8S-ClusterTest-SupplementalPluginsNestedStackSupplementalPluginsNestedStack-1Y25214IX1XHB User Initiated
INFRA-CDK-K8S-ClusterTest-SupplementalPluginsNestedStackSupplementalPluginsNestedStack-1Y25214IX1XHB |   0 | 8:26:28 PM | DELETE_IN_PROGRESS   | Custom::AWSCDK-EKS-KubernetesResource | SupplementalPlugins/ExternalSecrets/ServiceAccount/manifest-ServiceAccountServiceAccountResource/Resource/Default (ExternalSecretsServiceAccountmanifestServiceAccountServiceAccountResource54655F82)
...
operation in progress and is not stable (DELETE_IN_PROGRESS)
[20:52:06] Call failed: describeStacks({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack with id INFRA-CDK-K8S-ClusterTest does not exist (code=ValidationError)
[20:52:06] Stack INFRA-CDK-K8S-ClusterTest does not exist
[20:52:06] Call failed: describeStackEvents({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack [INFRA-CDK-K8S-ClusterTest] does not exist (code=ValidationError)
[20:52:06] Call failed: describeStackEvents({"StackName":"INFRA-CDK-K8S-ClusterTest"}) => Stack [INFRA-CDK-K8S-ClusterTest] does not exist (code=ValidationError)

 ✅  INFRA-CDK-K8S-ClusterTest: destroyed
[20:52:06] Reading cached notices from /Users/myuser/.cdk/cache/notices.json

  FAILED     integ/constructs/aws-eks/integ.xx-cluster-IntegTest/DefaultTest (undefined/us-east-2) 3058.427s
      Integration test failed: TypeError [ERR_STREAM_NULL_VALUES]: May not write null values to stream

Test Results:

Tests:    1 failed, 1 total
   --- Integration test metrics ---
Profile undefined + Region us-east-2 total time: 3058.428
  /Users/myuser/git/myorg/infra-cdk-k8s/test/integ/constructs/aws-eks/integ.xx-cluster.ts: 3058.428
Error: Some integration tests failed!
    at main (/Users/myuser/git/myorg/infra-cdk-k8s/node_modules/@aws-cdk/integ-runner/lib/index.js:10401:15)
👾 Task "integ" failed when executing "npx integ-runner --update-on-failed -vv test/integ/constructs/aws-eks/integ.xx-cluster.ts -vvvvv --parallel-regions=us-east-2" (cwd: /Users/myuser/git/myorg/infra-cdk-k8s)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
$ echo $?
1

Expected Behavior

I do not expect to see errors thrown during the integration tests - especially when the stacks themselves actually work and our deployments work.

Current Behavior

We’re seeing a strange error outputted about the stackName being invalid during some validation call… but yet the tests continue, and the stack itself seems fine.

Reproduction Steps

I don’t really know right now… we have multiple integration tests, and we only get this error on our largest test… this test takes 45m+ to complete (eks clusters + vpcs), and the codebase is very large so it’s not something I can easily share. I can say that the test has one stack with many nested stacks.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.100.0

Framework Version

No response

Node.js Version

18

OS

OSX and Linux

Language

TypeScript

Language Version

No response

Other information

I’m rolling back to our last known good release (2.93.0)… then I will roll forward one release at a time, but since the tests take nearly an hour to do each… it will take a while before I can track down which release specifically introduces the bug.