@piotrb yours is the only discussion I found of this problem. Just so you know, https://github.com/terraform-providers/terraform-provider-aws/pull/11463 should fix that.

This seems to have generally improved in the latest versions of the provider and TF itself … but some improvements would be great …

It doesn’t seem like the ordering causes the task definitions to need to be updated any more …

But when there is changes the diff is very nonsensical since it seems to be diffing against the raw order.

Could the provider not manage the order of the fields as they’re coming from aws and when they’re persisted to state? This would then just make the diff alphabetical and make the whole thing much more pleasant … right now its a pain to actually analyze the changes in the variables …

                  ~ environment       = [
                      - {
                          - name  = "AWS_REGION"
                          - value = "..."
                        },
                      - {
                          - name  = "PORT"
                          - value = "8001"
                        },
                      - {
                          - name  = "POD_SERVICE"
                          - value = "..."
                        },
                      - {
                          - name  = "POD_NAME"
                          - value = "..."
                        },
                        {
                            name  = "APP_NAME"
                            value = "..."
                        },
                      ~ {
                          ~ name  = "POD_ENVIRONMENT" -> "AWS_REGION"
                          ~ value = "..." -> "..."
                        },
                        {
                            name  = "LANG"
                            value = "en_US.UTF-8"
                        },
                      + {
                          + name  = "POD_ENVIRONMENT"
                          + value = "..."
                        },
                      + {
                          + name  = "POD_NAME"
                          + value = "..."
                        },
                      + {
                          + name  = "POD_SERVICE"
                          + value = "..."
                        },
                      + {
                          + name  = "PORT"
                          + value = "..."
                        },
                        {
                            name  = "SSM_PATHS"
                            value = "..."
                        },
                    ]

What was my actual change here? NONE … I didn’t add any env … I made other changes on the task definition …

It’s far from ideal but with the help of terraform 0.12 you can re-order the environment variables to match the order given by AWS.

1. create a Tf map with your env, in any order (because a map doesn’t have an order)

locals {
  unordered_env  = {
      ENV_A = "${var.my_var}"
      ENV_1 = "value_for_1"
    }
}

1. get the order from AWS, for a given task-definition run the following query

aws ecs describe-task-definition --task-definition $TASK_DEF | jq '[ .taskDefinition.containerDefinitions[0].environment[] | .name]'

1. copy the result into

locals {
   aws_order = [ .... ]
}

1. and the magic line which generate a list of maps in the right order

locals {
    environments_variables = [for order in local.aws_order : map(order, local.unordered_env[order])]
}

1. then use that list to generate your task definition, the list looks like this

environments_variables = [
    {
      "ENV_A" = "."............
    },
    {
      "ENV_1" = "................."
    },
]

1. (optional) it can be feed to the 0.11 old school template_file like this to to generate the json env list

data "template_file" "environments_variables" {
  template = " { \"name\" : \"$${name}\", \"value\": \"$${value}\" }"
  count    = length(var.environments_variables)
  vars = {
    value = "${element(values(var.environments_variables[count.index]), 0)}"
    name  = "${element(keys(var.environments_variables[count.index]), 0)}"
  }
}

data "template_file" "environments_variables_json" {
  template = "[  $${value}  ]"
  vars = {
    value = "${join(",", data.template_file.environments_variables.*.rendered)}"
  }
}

it doesn’t work with 0.11 because the for each interpolation syntax is from 0.12 only

you need to re-run the aws cli command and update the aws_order list each time you add/remove an env

@tamsky is correct. However, for clarity, this should still be considered a bug since it renders a confusing diff - as evidenced by this thread. 😃

Can we please have an update, we are having lots of issues with this bug @s-maj

@nhooey In my last comment, I tried to explain that the order of ENVIRONMENT elements in terraform plan output is misleading.

The aws_provider is 100% properly canonicalizing all ENVIRONMENT variable lists before calculating diffs. I even link to the code where that happens.

At the same time, terraform plan output is not and does not canonicalize that struct before emitting it, giving an appearance of being the source of a diff, when, if only order is different, it is not a diff.

It is my strong opinion that there is something else in your ECS task resource’s definition that is different.

Have you checked the provider’s output under TF_LOG=DEBUG ?

So it looks like there’s also issues with volumes too:

      - volume {
          - name = "gocd_data" -> null

          - docker_volume_configuration {
              - autoprovision = false -> null
              - driver        = "local" -> null
              - driver_opts   = {
                  - "device" = ":/"
                  - "o"      = "addr=fs-xxxxxxxx.efs.eu-west-2.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
                  - "type"   = "nfs"
                } -> null
              - labels        = {} -> null
              - scope         = "task" -> null
            }
        }
      + volume {
          + name = "gocd_data"

          + docker_volume_configuration {
              + autoprovision = false
              + driver        = "local"
              + driver_opts   = {
                  + "device" = ":/"
                  + "o"      = "addr=fs-xxxxxxxx.efs.eu-west-2.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
                  + "type"   = "nfs"
                }
              + scope         = "task"
            }
        }
      - volume {
          - name = "gocd_home" -> null

          - docker_volume_configuration {
              - autoprovision = false -> null
              - driver        = "local" -> null
              - driver_opts   = {
                  - "device" = ":/"
                  - "o"      = "addr=fs-xxxxxxxx.efs.eu-west-2.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
                  - "type"   = "nfs"
                } -> null
              - labels        = {} -> null
              - scope         = "task" -> null
            }
        }
      + volume {
          + name = "gocd_home"

          + docker_volume_configuration {
              + autoprovision = false
              + driver        = "local"
              + driver_opts   = {
                  + "device" = ":/"
                  + "o"      = "addr=fs-xxxxxxxx.efs.eu-west-2.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
                  + "type"   = "nfs"
                }
              + scope         = "task"
            }
        }

However, if there are no other changes the volumes don’t show changes either. It appears that if there are other changes then the volumes get added to the diff also. i.e. if the container_definitions have changes, then so does volumes.

Hello,

Just wondering if there’s any update on this? This is unfortunately causing our ECS Task Definition to be replaced every time despite zero changes.

The ordering once it gets to AWS certainly seems very random!

It’s possible to work around this by making sure your local task definition has the same order as stored on Amazon, which can be queried using the AWS cli (or just copy, paste, and sanitise the json that Terraform returns when planning). The order will remain the same until you add a new variable key-pair. Annoying, but it’s the only way to avoid your plans getting junked.

Improvements to this difference handling have been merged and will release with version 2.68.0 of the Terraform AWS Provider, later this week. Thanks to @jbergknoff-rival for the implementation. 👍

Is there a plan to add a feature that lets the user say “Ignore the order of JSON lists for this particular JSON field value”, so the AWS ENVIRONMENT JSON list doesn’t get detected as a diff when AWS chooses a stupid, arbitrary order that is different from what was submitted by Terraform?

In my case, the ECS tasks are getting recreated every time because AWS re-orders JSON lists unpredictably, but the reordering is static.

It’s hard to see what the course of action is from this ticket.

Suggestion:

Use the DEBUG output from the provider to determine the actual source of your diffs.

tl;dr summary:

Inconsistent order between environment objects is highly unlikely to be the cause of resource-level diffs.

diffs shown in plan output are not canonicalized. Therefore, there can be visual differences in their element ordering.

my problem/fix summary:

I lacked default values for my healthcheck:s. Adding the following values resolved my persistent diffs issue:

    interval: 30
    retries: 3
    timeout: 5

I’d suggest these missing values should not have caused diffs. Perhaps there’s an existing feature request? If someone knows of one, please mention/link here.

---

Debug journal:

terraform: v0.11.14 with plugin.terraform-provider-aws_v2.20.0_x4:.

My first guess was that I had diffs because the value of this field is an opaque JSON string, and differences in object sort returned by the AWS API would be the cause of diffs.

Turns out, there’s an entire go file dedicated to comparing ECS task definitions:

• https://github.com/terraform-providers/terraform-provider-aws/blob/v2.20.0/aws/ecs_task_definition_equivalency.go#L79-L82

That looks good, and the tests look good; it appears to effectively test that un-ordered environment objects compare as “equal to” ordered environment objects.

The tested function is actually being called, so it’s active code:

• https://github.com/terraform-providers/terraform-provider-aws/blob/v2.20.0/aws/resource_aws_ecs_task_definition.go#L81

I’m now convinced object order is not my problem.

Now I want to see the canonicalized JSON and check it for diffs.

Checking TF_LOG=DEBUG output, I was able to see the canonicalized First: and Second: DEBUG values and was able to compare them for actual diffs.

No surprise, there were actual diffs.

My issue was due to not including any values for a few healthcheck: defaults which the AWS API happily inserts.

There is nice a nice function https://github.com/terraform-providers/terraform-provider-aws/blob/fae04dfedfbd653d6a0bdbcc5d7c04f3d54e3048/aws/ecs_task_definition_equivalency.go#L56 used to reorder (diff suppression) task definition during diff but it’s used only there. All values go raw to the state https://github.com/terraform-providers/terraform-provider-aws/blob/fae04dfedfbd653d6a0bdbcc5d7c04f3d54e3048/aws/resource_aws_ecs_task_definition.go#L265 https://github.com/terraform-providers/terraform-provider-aws/blob/401dc017401659015fa0afdde5336b891c695fe6/aws/structure.go#L621

@radeksimko added task definition migration from sha to json (thanks million for that) plus further fixes. I was wondering if you could help us again to get this fixed 😃