aws-cdk: (aws-ecs): can't update ECS cluster because of "capacity provider is in use"

A previously (before 1.108.0) created ECS cluster with FARGATE and FARGATE_SPOT capacity providers can’t be updated anymore because of this error:

Error occurred during operation 'putClusterCapacityProviders SDK error: The specified capacity provider is in use and cannot be removed.

Reproduction Steps

Create an ECS cluster with

capacityProviders: ["FARGATE", "FARGATE_SPOT"],

with a CDK version prior to 1.108.0

Then upgrade CDK to latest version (1.110.1), try to update the stack to

enableFargateCapacityProviders: true,

(because the old property is deprecated)

What did you expect to happen?

Capacity providers should be set correctly and cluster should be updated

What actually happened?

Resource handler returned message: “Error occurred during operation ‘putClusterCapacityProviders SDK error: The specified capacity provider is in use and cannot be removed. (Service: AmazonECS; Status Code: 400; Error Code: ResourceInUseException; Request ID: ee1d2754-517d-4148-bbb1-25d0f8a1aa88; Proxy: null)’.” (RequestToken: 3d87b953-a468-bffc-d442-862a457b1f9b, HandlerErrorCode: GeneralServiceException)

Environment

  • CDK CLI Version : 1.110.1
  • Framework Version: 1.110.1
  • Node.js Version: v14.17.0
  • OS : MacOS 11.4
  • Language (Version): Typescript 4.1.2

Other

Probably caused by https://github.com/aws/aws-cdk/commit/6b2d0e0c867651cd632be9ca99c6e342fb3c1067


This is 🐛 Bug Report

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 12
  • Comments: 15 (6 by maintainers)

Most upvoted comments

The issue was introduced with this bug fix: https://github.com/aws/aws-cdk/pull/15012/

Unfortunately, we are now between a rock and a hard place, because if we just revert that change, customers who have deployed with 1.108.0 and up could run into this same issue.

In the meantime you can apply this workaround to your cluster. This is the best I could come up with, there might be a simpler way.

import * as cdk from '@aws-cdk/core';
import * as ecs from '@aws-cdk/aws-ecs';

export class MyStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const cluster = new ecs.Cluster(this, 'MyCluster', {
      enableFargateCapacityProviders: true,
    });

    // Add back the old method of specifying capacity providers
    const cfnCluster = cluster.node.defaultChild as ecs.CfnCluster;
    cfnCluster.capacityProviders = ['FARGATE', 'FARGATE_SPOT'];

    // Remove the new method of specifying capacity providers. 
    cdk.Aspects.of(this).add(new MyAspect());
  }
}

class MyAspect implements cdk.IAspect {
  public visit(node: cdk.IConstruct): void {
    if (node instanceof ecs.CfnClusterCapacityProviderAssociations) {
      // IMPORTANT: The id supplied here must be the same as the id of your cluster. Don't worry, you won't remove the cluster.  
      node.node.scope?.node.tryRemoveChild('MyCluster');
    }
  }
}

Note that a simple cluster.node.tryRemoveChild('MyCluster') doesn’t work here, which is why I used an Aspect to remove the CfnClusterCapacityProviderAssociations resource. I am not sure exactly why this is. I think it could be because the resource was added with an Aspect here.

Looks like this is a breaking change and we should try to fix this as soon as possible. I am going to try to repro it.

Thanks @madeline-k !

If anyone is interested in the Pythonic version of this workaround using CDK 2.x:

from aws_cdk import (
    Stack,
    aws_ecs as ecs
)
import jsii

@jsii.implements(cdk.IAspect)
class MyAspect:
  def visit(self, node):
    if isinstance(node, ecs.CfnClusterCapacityProviderAssociations):
      if node.node.scope:
        node.node.scope.node.try_remove_child('MyCluster')
from aws_cdk import (
    Stack,
    Aspects,
    aws_ecs as ecs
)

class MyStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
        cluster = ecs.Cluster(self, 'MyCluster', enable_fargate_capacity_providers=True)
        cfnCluster = cluster.node.default_child
        cfnCluster.capacity_providers = ['FARGATE', 'FARGATE_SPOT']
        Aspects.of(self).add(MyAspect())

I believe this is because:

ClusterCapacityProviderAssociations has specified both FARGATE and FARGATE_SPOT capacity provider but FARGATE capacity provider is in use by AWS::ECS::Service so you can’t delete ClusterCapacityProviderAssociations before the fargate service. Instead you need delete fargate service before the ClusterCapacityProviderAssociations. Generally you could just ensure the dependency using

service.node.addDependency(cfnassociation)

However, given the CfnClusterCapacityProviderAssociation is created from an Aspect, you can’t use escape hatches to ensure the dependency.

The only solution that comes to my mind is using another Aspect.

Consider the full sample below:

class EnsureCapacityProviderAssociationsDependency implements IAspect {
  private service: ecs.BaseService[];

  constructor(service: ecs.BaseService[]) {
    this.service = service;
  }

  public visit(node: IConstruct): void {
    if (node instanceof ecs.CfnClusterCapacityProviderAssociations) {
      console.log('found AWS::ECS::ClusterCapacityProviderAssociations')
      const cfnassociation = node as ecs.CfnClusterCapacityProviderAssociations
      this.service.forEach(s => {
        s.node.addDependency(cfnassociation)
        // cfnassociation.node.addDependency(s)
        console.log('added dependency to ' + s.node.id)
      })
    }
  }
}

export class DummyStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const vpc = getDefaultVpc(this);

    // ECS Cluster
    const cluster = new ecs.Cluster(this, 'EcsCluster', {
      vpc,
      enableFargateCapacityProviders: true,
    });

    // ECS Task Definition
    const ecsTaskDef = new ecs.FargateTaskDefinition(this, 'EcsTaskDef', {
    });

    ecsTaskDef.addContainer('web', {
      image: ecs.ContainerImage.fromRegistry('amazon/amazon-ecs-sample'),
    });

    // ECS Service
    const service = new ecs.FargateService(this, 'EcsService', {
      cluster,
      taskDefinition: ecsTaskDef,
      capacityProviderStrategies: [
        {
          capacityProvider: 'FARGATE',
          base: 0,
          weight: 1,
        },
      ],
    });

    // we need ensure the dependency but CfnClusterCapacityProviderAssociations is added by Aspects
    // https://github.com/aws/aws-cdk/blob/9946ab03672bf6664e8ec95a81ddb67c3bb2f63b/packages/aws-cdk-lib/aws-ecs/lib/cluster.ts#L1380C29-L1380C67
    // so we need another Aspect to fix it
    Aspects.of(cluster).add(new EnsureCapacityProviderAssociationsDependency([service]))
  }
}

On cdk synth you would see the service dependsOn the association

 "EcsService81FC6EF6": {
   "Type": "AWS::ECS::Service",
   "Properties": {
    "CapacityProviderStrategy": [
     {
      "Base": 0,
      "CapacityProvider": "FARGATE",
      "Weight": 1
     }
    ],
   ....
   },
   "DependsOn": [
    "EcsCluster72B17558",  // <--- CfnCapacityProviderAssociations here
    "EcsTaskDefTaskRole4E058A8F"
   ],

With the ensured dependency, service would always be destroyed “before” the CfnCapacityProviderAssociation so you can unblock it.

 % npx cdk destroy
found AWS::ECS::ClusterCapacityProviderAssociations
added dependency to EcsService
Are you sure you want to delete: dummy-stack17 (y/n)? y
dummy-stack17: destroying... [1/1]

 ✅  dummy-stack17: destroyed

Let me know if it works for you.

Thanks Madeline! Any idea if addAsgCapacityProvider regressed as a whole? I’m trying it on a brand new ECS cluster and I keep hitting the same exception

Resource handler returned message: "Out of retries. Last encountered error was: The specified capacity provider is in use and cannot be removed. (Service: AmazonECS; Status Code: 400; Error Code: ResourceInUseException; Request ID:

I have same issue with cdk destroy

I get The specified capacity provider is in use and cannot be removed when cdk destroy brand new stacks with CDK v2.13.0 and add leverage ecs.Cluster capacityProviders: ["FARGATE", "FARGATE_SPOT"]

Is this also known?