cdk-eks-blueprints: 1.6.0 to 1.6.1 upgrade breaking changes: AWS::KMS::Key used for secret encryption can not be replaced

Describe the bug

Having exciting eks cluster provisioned with cdk blueprint 1.6.0 & aws cdk 2.66.1 and using .useDefaultSecretEncryption(true) which is set by default if not present, fails during cdk deploy when cdk blueprint is upgraded to 1.6.1 & aws cdk 2.72.0 with following error:

 UPDATE_FAILED        | Custom::AWSCDK-EKS-Cluster            |  awskeyresourceID
Received response status [FAILED] from custom resource. Message returned: Cannot update cluster encryption configuration

Logs: /aws/lambda/eks-cluster-name-awscdk-OnEventHandler42BEBAE0-tg5FN0dg7Szg

at ClusterResourceHandler.onUpdate (/var/task/cluster.js:1:2271)
at ClusterResourceHandler.onEvent (/var/task/common.js:1:680)
at Runtime.onEvent [as handler] (/var/task/index.js:1:1434)
at Runtime.handleOnceNonStreaming (/var/runtime/Runtime.js:74:25) (RequestId: 1fbd2f3d-2d02-405b-89f5-2eacac7bfffb)

CloudWatch log: Logs: /aws/lambda/eks-cluster-name-awscdk-OnEventHandler42BEBAE0-tg5FN0dg7Szg is having this error:


{
    "errorType": "Error",
    "errorMessage": "Cannot update cluster encryption configuration",
    "stack": [
        "Error: Cannot update cluster encryption configuration",
        "    at ClusterResourceHandler.onUpdate (/var/task/cluster.js:1:2271)",
        "    at ClusterResourceHandler.onEvent (/var/task/common.js:1:680)",
        "    at Runtime.onEvent [as handler] (/var/task/index.js:1:1434)",
        "    at Runtime.handleOnceNonStreaming (/var/runtime/Runtime.js:74:25)"
    ]
}

This is output of cdk diff which show default AWS KMS key need to be destroyed and recreated.

cdk-diff

Look this issue which describes cannot change EKS cluster encryption config https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1234

Expected Behavior

Upgrading cdk blueprint from 1.6.0 & aws cdk 2.66.1 to 1.6.1 & aws cdk 2.72.0 should upgrade without problem exciting eks cluster created with default setting for .useDefaultSecretEncryption(true)

Current Behavior

Running cdk deploy fails:

 UPDATE_FAILED        | Custom::AWSCDK-EKS-Cluster            |  awskeyresourceID
Received response status [FAILED] from custom resource. Message returned: Cannot update cluster encryption configuration

Logs: /aws/lambda/eks-cluster-name-awscdk-OnEventHandler42BEBAE0-tg5FN0dg7Szg

at ClusterResourceHandler.onUpdate (/var/task/cluster.js:1:2271)
at ClusterResourceHandler.onEvent (/var/task/common.js:1:680)
at Runtime.onEvent [as handler] (/var/task/index.js:1:1434)
at Runtime.handleOnceNonStreaming (/var/runtime/Runtime.js:74:25) (RequestId: 1fbd2f3d-2d02-405b-89f5-2eacac7bfffb)

CloudWatch log: Logs: /aws/lambda/eks-cluster-name-awscdk-OnEventHandler42BEBAE0-tg5FN0dg7Szg is having this error:


{
    "errorType": "Error",
    "errorMessage": "Cannot update cluster encryption configuration",
    "stack": [
        "Error: Cannot update cluster encryption configuration",
        "    at ClusterResourceHandler.onUpdate (/var/task/cluster.js:1:2271)",
        "    at ClusterResourceHandler.onEvent (/var/task/common.js:1:680)",
        "    at Runtime.onEvent [as handler] (/var/task/index.js:1:1434)",
        "    at Runtime.handleOnceNonStreaming (/var/runtime/Runtime.js:74:25)"
    ]
}

Reproduction Steps

Install eks cluster using cdk blueprint 1.6.0 and aws cdk 2.66.1 and then try to upgrade cdk blueprint to 1.6.1 and aws cdk to 2.72.0 and try to run cdk deploy will reproduce this

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.72.0 (build 397e46c)

EKS Blueprints Version

1.6.1

Node.js Version

v16.17.0

Environment details (OS name and version, etc.)

Mac OS 12.6 Monterey - M1 Max

Other information

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 17 (9 by maintainers)

Most upvoted comments

This lead me to a rollback failed, how did you proceed from that?

edit: was able to continue rollback while ignoring the cluster, ran deploy again on 1.7.0, looks like its working

@Feder1co5oave i am not suggesting workaround, you probably mean workaround and PR from @aliaksei-ivanou