terraform-provider-aws: aws_ecs_service InvalidParameterException: Creation of service was not idempotent

This issue was originally opened by @simoneroselli as hashicorp/terraform#16635. It was migrated here as a result of the provider split. The original body of the issue is below.


Terraform Version

v0.10.8

Hi,

terraform is failing on modify the “placement strategy” for ECS/Service resources. Since such value can only be set at Service creation time, the expected beahviour would be “destroy 1, add 1”, like the terraform plan correctly reports. However on terraform apply it fails.

Fail Output

Error: Error applying plan:

1 error(s) occurred:

  • module.<my_module>.aws_ecs_service.<my_resource_name>: 1 error(s) occurred:

  • aws_ecs_service.main: InvalidParameterException: Creation of service was not idempotent. status code: 400, request id: xxxxxxxxxxxxxxxx “…”

Expected Behavior

destroy service, add service.

Actual Behavior

Failure of terraform without modification.

Steps to Reproduce

  1. Define a ECS/service with a placement strategy and apply
  2. Change the placement strategy values to something else
  3. terraform plan Plan: 1 to add, 0 to change, 1 to destroy.
  4. terraform apply InvalidParameterException: Creation of service was not idempotent

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 69
  • Comments: 33 (2 by maintainers)

Commits related to this issue

Most upvoted comments

Ran into the same issue, removing

lifecycle {
  create_before_destroy = true
}

From the ecs resource as @oanasabau noted worked for us too.

I can confirm, I have the same issue. The workaround is to remove the service manually.

Adding to the chorus suggesting that (with terraform 0.13.4, aws provider 3.11.0) this issue persists when replacing a service in place.

  • The first tf apply will complete, the old resource will (eventually) be deposed and the creation will fail
module.foo.aws_ecs_service.service[0]: Destruction complete after 6m18s

Error: InvalidParameterException: Creation of service was not idempotent. "foo"
  • The second attempt at applying this will result in a traditional success

As such it’s my belief there is some kind of a race condition or internal state issue that is not allowing the subsequent creation of a service after the API returns from the deletion. Is there a means of injecting a manual “sleep” here? We would benefit from resolving this so our CI/CD pipeline can manage this transition rather than requiring “babysitting” through two deploys, with increased downtime.

As it turns out in our case the problem was caused by create_before_destroy flag from the resource’s lifecycle policy. After removing that the terraform apply succeeded.

Getting this on TF v0.9.11 without placement strategy configured.

The best way I found is to change the name ( +1 @davidminor ) leaving however the lifecycle create_before_destroy = true

In this way a new service is created without interrupting the service and the old one is deposed only when the new one is active.

We ran into issue today as well:

	* failed creating ECS service (x-svc): InvalidParameterException: Creation of service was not idempotent.

In our case, we have another layer (pulumi), but quite sure this is due to the terraform provider. no “create_before_destroy” set explicitly by us, and we do have an ALB + target group pointing at service:

        loadBalancers: [
          {
            targetGroupArn: targetGroup.arn,
            containerName: containerName,
            containerPort: containerPort,
          },
        ],

EDIT: the solution here was to remove the service manually on AWS side also. I suspect root cause is that terraform was defensive and bailed when service already existed. Maybe a better error message eg “Service already exists - please remove or import properly” would communicate the problem/solution straight away and could be the fix for this issue.

We ran into this today. We weren’t setting any lifecycle rules on the aws_ecs_service resource, but repeatedly applies would fail due to what looks like a race condition between destruction of the aws_ecs_service and the re-creation of it.

We attempted all of the solutions listed above:

  • deleted the service manually and re-ran: same problem
  • re-imported the service into terraform state and re-ran: same problem

I randomly got it working by just… trying again. What’s interesting is the same plan/apply was run without issue in our lower environments: identical configuration, identical changeset, but those worked flawlessly.

I suspect it was pure luck that it worked fine twice before, and again when it worked after just trying again.

Terraform version: 1.1.4 AWS Provider version: 4.8.0

Deleting the service and running terraform apply fixed it for me.

due to this issue and missing name_prefix I tried:

name = “myservice_${replace(timestamp(), “:”, “-”)}”

and create_before_destroy in the service. This helps particulary, but is does not have the effect of beeing zero downtime deployment. The service is now beeing recreated every terraform run.

Ran into it today. Indeed, if create_before_destroy is used, we need name_prefix instead of name.

Has anyone found a workaround of generating a new name? Tried to use random_id but not sure what to use for “keepers” section.

@bkate i had the same issue, terraform deleted the service but after launched this error: “InvalidParameterException: Creation of service was not idempotent.”

i rerunning the apply and that works fine

I have same issue but got solved by rerunning terraform apply on the same resource. I think terraform cannot destroy and create the service at same time so it need a two steps resource apply or just remove the lifecycle would also solve the issue.

I think the same issue applies if you have the load_balancer, and likely, the target_group_arn, to the ECS service as well, as those settings can only be applied when creating the service.

In my use case (just the load_balancer block, no ordered_placement_strategy block), the service gets provisioned properly, but its state never gets recorded, not even partially. So, in subsequent TF runs, it said that it wants to add a brand new ECS service, but it would error out with the same “Creation of service was not idempotent” message.