terraform-provider-aws: aws_ecs_service InvalidParameterException: Creation of service was not idempotent
This issue was originally opened by @simoneroselli as hashicorp/terraform#16635. It was migrated here as a result of the provider split. The original body of the issue is below.
Terraform Version
v0.10.8
Hi,
terraform is failing on modify the “placement strategy” for ECS/Service resources. Since such value can only be set at Service creation time, the expected beahviour would be “destroy 1, add 1”, like the terraform plan correctly reports. However on terraform apply it fails.
Fail Output
Error: Error applying plan:
1 error(s) occurred:
-
module.<my_module>.aws_ecs_service.<my_resource_name>: 1 error(s) occurred:
-
aws_ecs_service.main: InvalidParameterException: Creation of service was not idempotent. status code: 400, request id: xxxxxxxxxxxxxxxx “…”
Expected Behavior
destroy service, add service.
Actual Behavior
Failure of terraform without modification.
Steps to Reproduce
- Define a ECS/service with a placement strategy and apply
- Change the placement strategy values to something else
terraform planPlan: 1 to add, 0 to change, 1 to destroy.terraform applyInvalidParameterException: Creation of service was not idempotent
About this issue
- Original URL
- State: open
- Created 7 years ago
- Reactions: 69
- Comments: 33 (2 by maintainers)
Ran into the same issue, removing
From the ecs resource as @oanasabau noted worked for us too.
I can confirm, I have the same issue. The workaround is to remove the service manually.
Adding to the chorus suggesting that (with
terraform 0.13.4, aws provider3.11.0) this issue persists when replacing a service in place.tf applywill complete, the old resource will (eventually) be deposed and the creation will failAs such it’s my belief there is some kind of a race condition or internal state issue that is not allowing the subsequent creation of a service after the API returns from the deletion. Is there a means of injecting a manual “sleep” here? We would benefit from resolving this so our CI/CD pipeline can manage this transition rather than requiring “babysitting” through two deploys, with increased downtime.
As it turns out in our case the problem was caused by create_before_destroy flag from the resource’s lifecycle policy. After removing that the terraform apply succeeded.
Getting this on TF v0.9.11 without placement strategy configured.
The best way I found is to change the name ( +1 @davidminor ) leaving however the lifecycle create_before_destroy = true
In this way a new service is created without interrupting the service and the old one is deposed only when the new one is active.
We ran into issue today as well:
In our case, we have another layer (pulumi), but quite sure this is due to the terraform provider. no “create_before_destroy” set explicitly by us, and we do have an ALB + target group pointing at service:
EDIT: the solution here was to remove the service manually on AWS side also. I suspect root cause is that terraform was defensive and bailed when service already existed. Maybe a better error message eg “Service already exists - please remove or import properly” would communicate the problem/solution straight away and could be the fix for this issue.
We ran into this today. We weren’t setting any lifecycle rules on the
aws_ecs_serviceresource, but repeatedly applies would fail due to what looks like a race condition between destruction of theaws_ecs_serviceand the re-creation of it.We attempted all of the solutions listed above:
I randomly got it working by just… trying again. What’s interesting is the same plan/apply was run without issue in our lower environments: identical configuration, identical changeset, but those worked flawlessly.
I suspect it was pure luck that it worked fine twice before, and again when it worked after just trying again.
Terraform version: 1.1.4 AWS Provider version: 4.8.0
Deleting the service and running
terraform applyfixed it for me.due to this issue and missing name_prefix I tried:
and create_before_destroy in the service. This helps particulary, but is does not have the effect of beeing zero downtime deployment. The service is now beeing recreated every terraform run.
Ran into it today. Indeed, if create_before_destroy is used, we need name_prefix instead of name.
Has anyone found a workaround of generating a new name? Tried to use random_id but not sure what to use for “keepers” section.
@bkate i had the same issue, terraform deleted the service but after launched this error: “InvalidParameterException: Creation of service was not idempotent.”
i rerunning the apply and that works fine
I have same issue but got solved by rerunning terraform apply on the same resource. I think terraform cannot destroy and create the service at same time so it need a two steps resource apply or just remove the lifecycle would also solve the issue.
I think the same issue applies if you have the load_balancer, and likely, the target_group_arn, to the ECS service as well, as those settings can only be applied when creating the service.
In my use case (just the load_balancer block, no ordered_placement_strategy block), the service gets provisioned properly, but its state never gets recorded, not even partially. So, in subsequent TF runs, it said that it wants to add a brand new ECS service, but it would error out with the same “Creation of service was not idempotent” message.