cdk-eks-blueprints: cdk deploy: State functionActiveV2 not found.

Describe the bug

cdk version: aws-cdk@2.66.1 node version v19.8.1 region: us-east-1

10:16:53 PM | CREATE_FAILED        | Custom::AWSCDK-EKS-HelmChart          | magicapi-eks/chart...r/Resource/Default
Received response status [FAILED] from custom resource. Message returned: StateNotFoundError: State functionActiveV2 not found.
at constructor.loadWaiterConfig (/var/runtime/node_modules/aws-sdk/lib/resource_waiter.js:196:32)
at new constructor (/var/runtime/node_modules/aws-sdk/lib/resource_waiter.js:64:10)
at features.constructor.waitFor (/var/runtime/node_modules/aws-sdk/lib/service.js:271:18)
at Object.defaultInvokeFunction [as invokeFunction] (/var/task/outbound.js:1:826)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async invokeUserFunction (/var/task/framework.js:1:2149)
at async onEvent (/var/task/framework.js:1:365)
at async Runtime.handler (/var/task/cfn-response.js:1:1543) (RequestId: f134e50c-5ed8-4bc2-a583-ea457527454c)

Expected Behavior

the deploy should succeed

Current Behavior

cdk deploy is failing

Reproduction Steps

Just follow the steps here https://aws-quickstart.github.io/cdk-eks-blueprints/getting-started/

Not sure why is it not working!

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.66.1

EKS Blueprints Version

1.6.0

Node.js Version

v19.8.1

Environment details (OS name and version, etc.)

OSX

Other information

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 19 (6 by maintainers)

Commits related to this issue

Most upvoted comments

@Sean-McAuliffe we would certainly consider shipping you the machine, but we do most of our dev in the cloud. We can attempt to reproduce with node exporter in place, but it seems orthogonal to the issue. Is there anything particular that you can share with respect to your IAM setup? What permissions your identity has?

Also do you mind trying our patterns repo, e.g. our starter construct make pattern starter deploy ?

I ran through the getting started page from scratch, and was unable to reproduce this issue.

One possible problem could be that the region the stack is supposed to deploy to (us-east-2) is not the region bootstrapped when you run cdk bootstrap, since that will bootstrap the default region in your AWS credentials. If you want to make sure you are bootstrapping the correct region, you can run cdk bootstrap aws://<ACCOUNT-NUMBER>/us-east-2

Also, currently the Cloud AutoScaling addon requires a helm chart version, so that line from the getting started page should go new blueprints.addons.ClusterAutoScalerAddOn({ version: "9.29.0" }). This is temporary, and a solution should be merged within the next few days.

hey @zjaco13 - appreciate the reply. We were bootstrapped, promise. I think given that there’s 4-6 other issues open for this, we can treat it as a legit bug and not user error

@shapirov103 @Sean-McAuliffe I can confirm the exact same thing. Just following through the main guide. This is the one I used: https://aws-quickstart.github.io/cdk-eks-blueprints/getting-started/

Day before yesterday deployed a stack for the first time on “eu-central-1” successfully, tore it down, and tried spinning it up again yesterday, and received the following error. Tried updating, since there were updates coincidentally so I thought it might be related, but same error today.

Cloudstack error:

---
Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: Resource is not in the state functionActive at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:61:27) at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:61:8) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14) at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12) at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: 10f3132d-1390-46df-ae79-ff0555e2d3b1)

At

 10:49:17 AM | CREATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | eksblueprintawsloa...ntResource32DFFE2C

and

10:49:17 AM | CREATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | eksblueprintAwsAuthmanifest443E8097

  • NodeJS 18.6 (latest stable)
  • Both CDK 2.81, 2.82 and 2.83
  • CDK Blueprints 1.7 and 1.81
  • Updated other NPM packages as well after updating CDK to different versions.

And not completely related, but it did cause confusion on my end: whats the difference between https://aws-quickstart.github.io/cdk-eks-blueprints/getting-started/ and https://catalog.workshops.aws/eks-blueprints-for-cdk/en-US? Why are there two different guides for the same thing? The second one seems older, so eventually I stuck with the first one.

@shapirov103 - the get started page. we’re using it to introduce an intern to the CDK/EKS in general. deployed once OK in east-2, then we had the intern tear it down, replace the autoscale addon with Karpenter, and redeploy. First error was the above re: functionActiveV2, second one we got is this:

Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: Resource is not in the state functionActive

we tried with the most up-to-date CDK version, tried to downgrade to under 2.60, and added "@aws-cdk/customresources:installLatestAwsSdkDefault": true to his cdk.context.json file

Might be related to aws/aws-cdk#24358 (comment)

Try putting "@aws-cdk/customresources:installLatestAwsSdkDefault": true in your CDK context

I tried this and am still seeing the error. I also tried downgrading to an older cdk version as recommended in that issue.