aws-cdk: Code-Pipeline: update step delayed by missing files ('/root/.cdk/cache/notices.json', 'cdk.out/tree.json')

What is the problem?

The update step gets delayed, not aborted, by trying to open two missing files.

UpdateStack \| 29/57 \| 2:17:18 PM \| UPDATE_COMPLETE      \| AWS::CloudFormation::Stack  \| UpdateStack
--
[Container] 2022/02/28 14:14:24 Waiting for agent ping
[Container] 2022/02/28 14:14:25 Waiting for DOWNLOAD_SOURCE
[Container] 2022/02/28 14:14:26 Phase is DOWNLOAD_SOURCE
[Container] 2022/02/28 14:14:26 CODEBUILD_SRC_DIR=/codebuild/output/src677814557/src
[Container] 2022/02/28 14:14:26 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2022/02/28 14:14:26 Processing environment variables
[Container] 2022/02/28 14:14:27 No runtime version selected in buildspec.
[Container] 2022/02/28 14:14:29 Moving to directory /codebuild/output/src677814557/src
[Container] 2022/02/28 14:14:29 Configuring ssm agent with target id: codebuild:960179ab-f4cf-4c04-8582-5c6e4763d9fe
[Container] 2022/02/28 14:14:29 Successfully updated ssm agent configuration
[Container] 2022/02/28 14:14:29 Registering with agent
[Container] 2022/02/28 14:14:29 Phases found in YAML: 2
[Container] 2022/02/28 14:14:29  BUILD: 2 commands
[Container] 2022/02/28 14:14:29  INSTALL: 2 commands
[Container] 2022/02/28 14:14:29 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2022/02/28 14:14:29 Phase context status code:  Message: 
[Container] 2022/02/28 14:14:29 Entering phase INSTALL

[Container] 2022/02/28 14:14:29 Running command npm install -g aws-cdk
/usr/local/bin/cdk -> /usr/local/lib/node_modules/aws-cdk/bin/cdk
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/aws-cdk/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ aws-cdk@2.14.0
added 1 package from 1 contributor in 1.358s

[Container] 2022/02/28 14:14:34 Phase complete: INSTALL State: SUCCEEDED
[Container] 2022/02/28 14:14:34 Phase context status code:  Message: 
[Container] 2022/02/28 14:14:34 Entering phase PRE_BUILD
[Container] 2022/02/28 14:14:34 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2022/02/28 14:14:34 Phase context status code:  Message: 
[Container] 2022/02/28 14:14:34 Entering phase BUILD

[Container] 2022/02/28 14:14:34 Running command cdk -a . deploy UpdateStack --require-approval=never --verbose
CDK toolkit version: 2.14.0 (build 762d71b)
Command line arguments: {
  _: [ 'deploy' ],
  a: '.',
  app: '.',
  'require-approval': 'never',
  requireApproval: 'never',
  verbose: 1,
  v: 1,
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  all: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  logs: true,
  '$0': '/usr/local/bin/cdk',
  STACKS: [ 'UpdateStack' ],
  'S-t-a-c-k-s': [ 'UpdateStack' ]
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: '.',
  context: {},
  debug: false,
  assetMetadata: true,
  requireApproval: 'never',
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '*' ],
  lookups: true
}
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to <REGION>
Resolving default credentials
Looking up default account ID from STS
Default account ID: <ACCOUNT>
Setting "CDK_DEFAULT_ACCOUNT" environment variable to <ACCOUNT>
context: {
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
--app points to a cloud assembly, so we bypass synth

✨  Synthesis time: 0.17s

UpdateStack: deploying...
Retrieved account ID <ACCOUNT> from disk cache
Assuming role 'arn:aws:iam::<ACCOUNT>:role/cdk-hnb659fds-deploy-role-<ACCOUNT>-<REGION>'.
Waiting for stack CDKToolkit to finish creating or updating...
[0%] start: Publishing c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311:<ACCOUNT>-<REGION>
Retrieved account ID <ACCOUNT> from disk cache
Assuming role 'arn:aws:iam::<ACCOUNT>:role/cdk-hnb659fds-file-publishing-role-<ACCOUNT>-<REGION>'.
[0%] check: Check s3://cdk-hnb659fds-assets-<ACCOUNT>-<REGION>/c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311.json
Call failed: getBucketEncryption({"Bucket":"cdk-hnb659fds-assets-<ACCOUNT>-<REGION>"}) => Access Denied (code=AccessDenied)
[0%] debug: Could not read encryption settings of bucket 'cdk-hnb659fds-assets-<ACCOUNT>-<REGION>': uploading with default settings ("cdk bootstrap" to version 9 if your organization's policies prevent a successful upload or to get rid of this message).
[0%] upload: Upload s3://cdk-hnb659fds-assets-<ACCOUNT>-<REGION>/c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311.json
[100%] success: Published c7badc2e98dcd8e4a0d22c7c01aa415f83096259e42bca07d943f9c100de6311:<ACCOUNT>-<REGION>
UpdateStack: checking if we can skip deploy
UpdateStack: template has changed
UpdateStack: deploying...
Removing existing change set with name cdk-deploy-change-set if it exists
Attempting to create ChangeSet with name cdk-deploy-change-set to update stack UpdateStack
UpdateStack: creating CloudFormation changeset...
Initiated creation of changeset: arn:aws:cloudformation:<REGION>:<ACCOUNT>:changeSet/cdk-deploy-change-set/685a1ec5-7267-47f4-b859-30b5235c6e10; waiting for it to finish creating...
Waiting for changeset cdk-deploy-change-set on stack UpdateStack to finish creating...
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Changeset cdk-deploy-change-set on stack UpdateStack is still creating
Initiating execution of changeset arn:aws:cloudformation:<REGION>:<ACCOUNT>:changeSet/cdk-deploy-change-set/685a1ec5-7267-47f4-b859-30b5235c6e10 on stack UpdateStack
Execution of changeset arn:aws:cloudformation:<REGION>:<ACCOUNT>:changeSet/cdk-deploy-change-set/685a1ec5-7267-47f4-b859-30b5235c6e10 on stack UpdateStack has started; waiting for the update to complete...
Waiting for stack UpdateStack to finish creating or updating...
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS (User Initiated))
UpdateStack |  0/57 | 2:15:44 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack  | UpdateStack User Initiated
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
 0/57 Currently in progress: UpdateStack
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset40 (PipelineAssetsFileAsset405A6FEB45) 
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset31 (PipelineAssetsFileAsset315BF305B1) 
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset33 (PipelineAssetsFileAsset331EA5BEFF) 
UpdateStack |  0/57 | 2:16:37 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset30 (PipelineAssetsFileAsset30AA2C9CCE) 
UpdateStack |  0/57 | 2:16:38 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset32 (PipelineAssetsFileAsset32B2F439EA) 
UpdateStack |  0/57 | 2:16:38 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset44 (PipelineAssetsFileAsset44F4DEECBF) 
UpdateStack |  0/57 | 2:16:38 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset28 (PipelineAssetsFileAsset28D7887C5A) 
UpdateStack |  0/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset35 (PipelineAssetsFileAsset359C5D1AAE) 
UpdateStack |  0/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset34 (PipelineAssetsFileAsset34304F56C6) 
UpdateStack |  0/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset37 (PipelineAssetsFileAsset37A82313CE) 
UpdateStack |  1/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset31 (PipelineAssetsFileAsset315BF305B1) 
UpdateStack |  2/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset40 (PipelineAssetsFileAsset405A6FEB45) 
UpdateStack |  3/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset30 (PipelineAssetsFileAsset30AA2C9CCE) 
UpdateStack |  3/57 | 2:16:39 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset36 (PipelineAssetsFileAsset36A9A3262B) 
UpdateStack |  4/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset32 (PipelineAssetsFileAsset32B2F439EA) 
UpdateStack |  5/57 | 2:16:39 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset33 (PipelineAssetsFileAsset331EA5BEFF) 
UpdateStack |  5/57 | 2:16:40 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset23 (PipelineAssetsFileAsset23A5E4949B) 
UpdateStack |  5/57 | 2:16:40 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset29 (PipelineAssetsFileAsset2994CB4E12) 
UpdateStack |  5/57 | 2:16:40 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset48 (PipelineAssetsFileAsset4856493151) 
UpdateStack |  6/57 | 2:16:40 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset44 (PipelineAssetsFileAsset44F4DEECBF) 
UpdateStack |  7/57 | 2:16:40 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset35 (PipelineAssetsFileAsset359C5D1AAE) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack |  8/57 | 2:16:40 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset34 (PipelineAssetsFileAsset34304F56C6) 
UpdateStack |  9/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset28 (PipelineAssetsFileAsset28D7887C5A) 
UpdateStack | 10/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset37 (PipelineAssetsFileAsset37A82313CE) 
UpdateStack | 11/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset36 (PipelineAssetsFileAsset36A9A3262B) 
UpdateStack | 11/57 | 2:16:41 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset24 (PipelineAssetsFileAsset240D3F5D3D) 
UpdateStack | 11/57 | 2:16:41 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset39 (PipelineAssetsFileAsset3939491230) 
UpdateStack | 12/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset29 (PipelineAssetsFileAsset2994CB4E12) 
UpdateStack | 13/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset48 (PipelineAssetsFileAsset4856493151) 
UpdateStack | 14/57 | 2:16:41 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset23 (PipelineAssetsFileAsset23A5E4949B) 
UpdateStack | 14/57 | 2:16:42 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset16 (PipelineAssetsFileAsset161A3A5D08) 
UpdateStack | 14/57 | 2:16:42 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset14 (PipelineAssetsFileAsset14A3CA232F) 
UpdateStack | 15/57 | 2:16:43 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset24 (PipelineAssetsFileAsset240D3F5D3D) 
UpdateStack | 16/57 | 2:16:43 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset39 (PipelineAssetsFileAsset3939491230) 
UpdateStack | 16/57 | 2:16:43 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset27 (PipelineAssetsFileAsset278581B380) 
UpdateStack | 16/57 | 2:16:43 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset2 (PipelineAssetsFileAsset24D2D639B) 
UpdateStack | 16/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset43 (PipelineAssetsFileAsset43E66E3741) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset16 (PipelineAssetsFileAsset161A3A5D08) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset42 (PipelineAssetsFileAsset4293256298) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset25 (PipelineAssetsFileAsset25C2F636DD) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset41 (PipelineAssetsFileAsset4148861B0A) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset15 (PipelineAssetsFileAsset15B22B8501) 
UpdateStack | 17/57 | 2:16:44 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset26 (PipelineAssetsFileAsset26CD3C46ED) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 18/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset42 (PipelineAssetsFileAsset4293256298) 
UpdateStack | 19/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset15 (PipelineAssetsFileAsset15B22B8501) 
UpdateStack | 20/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset41 (PipelineAssetsFileAsset4148861B0A) 
UpdateStack | 21/57 | 2:16:46 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset26 (PipelineAssetsFileAsset26CD3C46ED) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 21/57 | 2:16:55 PM | UPDATE_IN_PROGRESS   | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset47 (PipelineAssetsFileAsset47191A3E0D) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 22/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset2 (PipelineAssetsFileAsset24D2D639B) 
UpdateStack | 23/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset14 (PipelineAssetsFileAsset14A3CA232F) 
UpdateStack | 24/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset27 (PipelineAssetsFileAsset278581B380) 
UpdateStack | 25/57 | 2:16:56 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset43 (PipelineAssetsFileAsset43E66E3741) 
UpdateStack | 26/57 | 2:16:57 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset47 (PipelineAssetsFileAsset47191A3E0D) 
UpdateStack | 27/57 | 2:16:58 PM | UPDATE_COMPLETE      | AWS::CodeBuild::Project     | Pipeline/Assets/FileAsset25 (PipelineAssetsFileAsset25C2F636DD) 
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
Stack UpdateStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS)
UpdateStack | 28/57 | 2:17:17 PM | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack  | UpdateStack 
Stack UpdateStack has completed updating

✅  UpdateStack

✨  Deployment time: 165.34s

Stack ARN:
arn:aws:cloudformation:<REGION>:<ARN>:stack/UpdateStack/f62b9db0-3e1d-11ec-a5dd-024bd07bc0e7

✨  Total time: 165.52s

Failed to load notices from cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json'
Error on request: Error: read ETIMEDOUT
Notices refreshed
Error on request: Error: read ETIMEDOUT
Failed to get tree.json file: Error: cdk.out/tree.json: ENOTDIR: not a directory, open 'cdk.out/tree.json'


[Container] 2022/02/28 14:33:13 Phase complete: BUILD State: SUCCEEDED

This error delays the Update Step by almost 16 minutes.

As source action we are using github action v2.

Do you have any idea whats going on here?

Regards

Reproduction Steps

start the pipeline using a github source action

What did you expect to happen?

no read timeouts for notice.json and tree.json

What actually happened?

a read timeout for notice.json and tree.json

CDK CLI Version

aws-cdk@2.15.0

Framework Version

No response

Node.js Version

14

OS

linux

Language

Typescript

Language Version

4.3.5

Other information

No response