terraform-provider-aws: Provider produced inconsistent result after apply ECS Fargate task
Community Note
- Please vote on this issue by adding a đź‘Ť reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Terraform CLI and Terraform AWS Provider Version
- Terraform 0.14.5
- aws provider 3.38.0
Affected Resource(s)
- aws_ecs_service
Terraform Configuration Files
Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.
# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp
Debug Output
Panic Output
Expected Behavior
Updated tags on ECS Fargate tasks images on 13 services
Actual Behavior
After updating 10 services the next one we got the error “Provider produced inconsistent result after apply”, weirdly enough this same code was run on different clusters at the same time on diferent pipelines multiple times before without any errors.
Error: Provider produced inconsistent result after apply
When applying changes to module.linux-app-1.aws_ecs_service.app["app11"],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
##[error]PowerShell exited with code '1'.
##[error]PowerShell wrote one or more lines to the standard error stream.
##[error]
Error: Provider produced inconsistent result after apply
When applying changes to module.linux-app-1.aws_ecs_service.app["app11"],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.
This caused a weirder error when running the pipeline for a second time “creation: InvalidParameterException: Creation of service was not idempotent.” So we commented out that service re-ran pipeline it applied, but the service was still in AWS but as INACTIVE. Had to manually destroy it and uncomment it, and the apply an the service went trough.
Error: error creating fargate-service service: error waiting for ECS service (fargate-app11-service) creation: InvalidParameterException: Creation of service was not idempotent.
on .terraform/modules/linux-app-1/modules/fargate-service/main.tf line 161, in resource "aws_ecs_service" "app":
161: resource "aws_ecs_service" "app" {
##[error]PowerShell exited with code '1'.
##[error]PowerShell wrote one or more lines to the standard error stream.
##[error]
Error: error creating fargate-app11-service service: error waiting for ECS service (fargate-app11-service) creation: InvalidParameterException: Creation of service was not idempotent.
on .terraform/modules/linux-app-1/modules/fargate-service/main.tf line 161, in resource "aws_ecs_service" "app":
161: resource "aws_ecs_service" "app" {
Steps to Reproduce
Don’t have a set of steps it was random, hasn’t come up since
Important Factoids
References
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 20
- Comments: 18 (13 by maintainers)
@anGie44 we’ve been running the fixed version for over 2 weeks without issue, just wanted to say thanks and that I owe you a beer/coffee/your beverage of choice!
For anyone else encountering this issue, update to the latest version of this module and enable
wait_for_steady_stateas a workaround.I’ve made a PR #23747 which has a partial workaround/fix for using wait_for_steady_state we’ve been running this for a few weeks and not had a failure once. Without wait_for_steady_state it still fails. It’ll be better if someone more familiar with the module digs deeper into this for a complete fix but for now this is a workaround.
Some more context on this, I’ve enabled the TF_LOG and found that the problem service is returning a status of INACTIVE so terraform sees this as an error and taints it which causes an error.
I’m guessing here but either 1. The service is failing to create properly or 2. Terraform isn’t waiting for the service to be destroyed before creating it (in our case we taint the service before running apply).
Here’s the except of the log:
Got it, it’s building now I’ll let you know how it goes.