aws-cdk: (aws-ecs): can't update ECS cluster because of "capacity provider is in use"
A previously (before 1.108.0) created ECS cluster with FARGATE and FARGATE_SPOT capacity providers can’t be updated anymore because of this error:
Error occurred during operation 'putClusterCapacityProviders SDK error: The specified capacity provider is in use and cannot be removed.
Reproduction Steps
Create an ECS cluster with
capacityProviders: ["FARGATE", "FARGATE_SPOT"],
with a CDK version prior to 1.108.0
Then upgrade CDK to latest version (1.110.1), try to update the stack to
enableFargateCapacityProviders: true,
(because the old property is deprecated)
What did you expect to happen?
Capacity providers should be set correctly and cluster should be updated
What actually happened?
Resource handler returned message: “Error occurred during operation ‘putClusterCapacityProviders SDK error: The specified capacity provider is in use and cannot be removed. (Service: AmazonECS; Status Code: 400; Error Code: ResourceInUseException; Request ID: ee1d2754-517d-4148-bbb1-25d0f8a1aa88; Proxy: null)’.” (RequestToken: 3d87b953-a468-bffc-d442-862a457b1f9b, HandlerErrorCode: GeneralServiceException)
Environment
- CDK CLI Version : 1.110.1
- Framework Version: 1.110.1
- Node.js Version: v14.17.0
- OS : MacOS 11.4
- Language (Version): Typescript 4.1.2
Other
Probably caused by https://github.com/aws/aws-cdk/commit/6b2d0e0c867651cd632be9ca99c6e342fb3c1067
This is 🐛 Bug Report
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 12
- Comments: 15 (6 by maintainers)
The issue was introduced with this bug fix: https://github.com/aws/aws-cdk/pull/15012/
Unfortunately, we are now between a rock and a hard place, because if we just revert that change, customers who have deployed with 1.108.0 and up could run into this same issue.
In the meantime you can apply this workaround to your cluster. This is the best I could come up with, there might be a simpler way.
Note that a simple
cluster.node.tryRemoveChild('MyCluster')doesn’t work here, which is why I used anAspectto remove theCfnClusterCapacityProviderAssociationsresource. I am not sure exactly why this is. I think it could be because the resource was added with an Aspect here.Looks like this is a breaking change and we should try to fix this as soon as possible. I am going to try to repro it.
Thanks @madeline-k !
If anyone is interested in the Pythonic version of this workaround using CDK 2.x:
I believe this is because:
ClusterCapacityProviderAssociationshas specified bothFARGATEandFARGATE_SPOTcapacity provider butFARGATEcapacity provider is in use byAWS::ECS::Serviceso you can’t deleteClusterCapacityProviderAssociationsbefore the fargate service. Instead you need delete fargate service before theClusterCapacityProviderAssociations. Generally you could just ensure the dependency usingHowever, given the CfnClusterCapacityProviderAssociation is created from an Aspect, you can’t use escape hatches to ensure the dependency.
The only solution that comes to my mind is using another Aspect.
Consider the full sample below:
On
cdk synthyou would see theservicedependsOn the associationWith the ensured dependency, service would always be destroyed “before” the
CfnCapacityProviderAssociationso you can unblock it.Let me know if it works for you.
Thanks Madeline! Any idea if addAsgCapacityProvider regressed as a whole? I’m trying it on a brand new ECS cluster and I keep hitting the same exception
Resource handler returned message: "Out of retries. Last encountered error was: The specified capacity provider is in use and cannot be removed. (Service: AmazonECS; Status Code: 400; Error Code: ResourceInUseException; Request ID:
I have same issue with
cdk destroyI get
The specified capacity provider is in use and cannot be removedwhencdk destroybrand new stacks with CDK v2.13.0 and add leverage ecs.ClustercapacityProviders: ["FARGATE", "FARGATE_SPOT"]Is this also known?