aws-cdk: eks: After upgrading to 2.80, imported cluster resources fail to deploy

Describe the bug

After upgrading CDK to 2.80, the EKS cluster creation role no longer can be assumed by separate handlers in stacks that import the EKS cluster as explained in https://github.com/aws/aws-cdk/issues/25674.

The suggested fix of importing the whole kubectlProvider results in Modifying service token is not allowed. error when trying to deploy the stack.

Expected Behavior

Kubernetes resources in the importing stack should still deploy in 2.80+ after adjusting the cluster importing as mentioned in https://github.com/aws/aws-cdk/issues/25674

Current Behavior

cdk8s chart in the importing stack fails to deploy after the upgrade.

StackNameRedacted: creating CloudFormation changeset… 2:20:09 PM | UPDATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | KubeClusterCommonK8sChartA8489958 Modifying service token is not allowed.

❌ StackNameRedacted failed: Error: The stack named StackNameRedacted failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed. at FullCloudFormationDeployment.monitorDeployment (/workdir/node_modules/aws-cdk/lib/index.js:397:10236) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.deployStack2 [as deployStack] (/workdir/node_modules/aws-cdk/lib/index.js:400:149977) at async /workdir/node_modules/aws-cdk/lib/index.js:400:135508

❌ Deployment failed: Error: The stack named StackNameRedacted failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed. at FullCloudFormationDeployment.monitorDeployment (/workdir/node_modules/aws-cdk/lib/index.js:397:10236) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.deployStack2 [as deployStack] (/workdir/node_modules/aws-cdk/lib/index.js:400:149977) at async /workdir/node_modules/aws-cdk/lib/index.js:400:135508

The stack named StackNameRedacted failed to deploy: UPDATE_ROLLBACK_COMPLETE: Modifying service token is not allowed.

Reproduction Steps

Export (app A, stack A)

const kubectlProvider = cluster.stack.node
  .findChild('@aws-cdk--aws-eks.KubectlProvider') as eks.KubectlProvider

new CfnOutput(scope, 'KubectlProviderRole', {
  exportName: 'KubectlRoleArn',
  value: kubectlProvider.roleArn,
});

new CfnOutput(scope, 'KubectlProviderHandlerRole', {
  exportName: 'KubectlHandlerRoleArn',
  value: kubectlProvider.handlerRole.roleArn,
});

const kubectlHandler = kubectlProvider.node.findChild('Handler') as lambda.IFunction;

new CfnOutput(scope, 'KubectlProviderHandler', {
  exportName: 'KubectlHandlerArn',
  value: kubectlHandler.functionArn,
});

// tried also
// new CfnOutput(scope, 'KubectlProviderHandler', {
//   exportName: 'KubectlHandlerArn',
//   value: kubectlProvider.serviceToken,
// });

Import (app B, stack B)

  const kubectlProvider = eks.KubectlProvider.fromKubectlProviderAttributes(
    scope,
    'KubectlProvider',
    {
      functionArn: Fn.importValue('KubectlHandlerArn'),
      handlerRole: iam.Role.fromRoleArn(
        scope,
        'HandlerRole',
        Fn.importValue('KubectlHandlerRoleArn')
      ),
      kubectlRoleArn: Fn.importValue('KubectlRoleArn'),
    }
  );

  const openIdConnectProvider = iam.OpenIdConnectProvider.fromOpenIdConnectProviderArn(
    scope,
    'KubeOidcProvider',
    Fn.importValue('KubernetesOidcProviderArn')
  );

  const kubectlSecurityGroupId = Fn.importValue('KubernetesControlPlaneSGId');
  return eks.Cluster.fromClusterAttributes(scope, 'KubeCluster', {
    clusterName,
    kubectlPrivateSubnetIds: Fn.split(',', Fn.importValue('KubernetesPrivateSubnetIds')),
    kubectlSecurityGroupId: kubectlSecurityGroupId,
    clusterSecurityGroupId: kubectlSecurityGroupId,
    vpc,
    openIdConnectProvider,
    kubectlLayer: new KubectlV25Layer(scope, 'kubectl'),
    kubectlProvider: kubectlProvider,
  });

Then trying to deploy the (already existing) stack B fails.

Possible Solution

No response

Additional Information/Context

Both of the stacks have been in use for a long time and the only change to the cluster importing was to replace kubectlRoleArn with kubectlProvider. They are part of bigger ckd apps and this problem affects multiple stacks in multiple importing apps.

CDK CLI Version

2.81.0 (build bd920f2)

Framework Version

No response

Node.js Version

v18.14.2

OS

Linux

Language

Typescript

Language Version

TypeScript (3.9.10)

Other information

Came across the following earlier issue with similar symptoms from back when eks was experimental in CDK: https://github.com/aws/aws-cdk/issues/6129.