terraform-provider-aws: "Objects changed outside of Terraform" shown wrongly for various resources

Is there any plan to do something about this? 🤔

Also affects aws_kms_key resources - the policy statement principals are changing order:

  ~ resource "aws_kms_key" "key" {
        id                       = ...
      ~ policy                   = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws:iam::XXXXXXXXXXXX:root",
                                "arn:aws:iam::YYYYYYYYYYYY:root",
                              + "arn:aws:iam::XXXXXXXXXXXX:root",
                            ]
                        }
                        # (4 unchanged elements hidden)
                    },
...

Also affects aws_wafv2_web_acl resources; it is unclear why, as the attributes list shows no changes. Applying the refresh plan has no effect — it will still be marked as changed in the next plan.

Observed on provider version v3.45.0.

I confirm it’s still the case with aws_wafv2_web_acl (provider version 3.69.0).

Terraform detected the following changes made outside of Terraform since the last “terraform apply”:

# aws_wafv2_web_acl.gogaille_cloudfront has changed
~ resource "aws_wafv2_web_acl" "some_resource" {
      id          = "731f3d3e-b5b9-48fe-9f91-3b73b7230839"
      name        = "some-name"
      tags        = {}
      # (6 unchanged attributes hidden)



      # (8 unchanged blocks hidden)
  }

I’m seeing an aws_iam_role policy as changed without even having a list being reordered, or any other change visibly pointed out. Terraform apparently includes # whitespace changes in the output when this happens.

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.identity.aws_iam_role.role has been changed
  ~ resource "aws_iam_role" "role" {
      ~ assume_role_policy    = jsonencode( # whitespace changes
            {
                Statement = [
                    {
                        # [snip for brevity]
                    },
                    {
                        # [snip for brevity]
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        # [snip for brevity]
    }

The aws_backup_vault resource is affected as well:

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_backup_vault.test has been changed
  ~ resource "aws_backup_vault" "test" {
        id              = "test"
        name            = "test"
      ~ recovery_points = 9 -> 10
        # (3 unchanged attributes hidden)
    }

This seems to affect anything with a policy. The principal list appears to come back in a non-deterministic order.

For reference, the original issue left open since June 2021 https://github.com/hashicorp/terraform/issues/28803

This also affects things that change outside of Terraform, but which you’ve told Terraform to ignore.

Two places we’ve seen that are (1) tags in a lifecycle ignore_changes list; and (2) when an ELB isn’t configured in Terraform to have listeners, because we have separate orchestration tools that add and remove them. In both of those cases, Terraform used to not show changes when those things changed behind the scenes, but now it does.

I’m not sure if that’s a provider side thing that fits with this ticket, or if the ignored-changes problem belongs with https://github.com/hashicorp/terraform/issues/28803.

@gilad9366 Please open an issue for aws_autoscaling_group if this is still an issue so that we can prioritize and get eyes on that specific problem.

@nantiferov I would love to hear back from you if you are still facing problems with drift using v4.50.0. Please see #23288.

According to this comment, since v1.2 (not released yet) it should became kinda better https://github.com/hashicorp/terraform/issues/28803#issuecomment-1072740861

Starting with v1.2, the goal for the refresh report is that only external changes which may have contributed to changes in the plan will be shown. This means in most cases, unused attributes changing outside of terraform will not show up in the normal plan output. If there are no changes in the plan, no external changes will be shown in the CLI at all. All refresh information is still stored within the plan, and if a user wants to see all external changes of resources in the CLI, a refresh-only plan can be used.

P.S. Btw, sed could be used to cut off this nice feature (macOS example), filter in realtime (i.e. you don’t have to wait till terraform finish), works with apply too, as stdin not affected.

terraform plan|sed -n '/Objects have changed outside of Terraform/,/─────────────────────────────────────────────────────────────────────────────/!p'

P.P.S. Also I wrote an article some time ago how to hide this stuff with python wrapper https://nklya.medium.com/fix-objects-have-changed-outside-of-terraform-with-invoke-wrapper-7b73fa2e99a0

I confirm it’s still the case with aws_wafv2_web_acl (provider version 3.69.0).

Terraform detected the following changes made outside of Terraform since the last “terraform apply”:

# aws_wafv2_web_acl.gogaille_cloudfront has changed
~ resource "aws_wafv2_web_acl" "some_resource" {
      id          = "731f3d3e-b5b9-48fe-9f91-3b73b7230839"
      name        = "some-name"
      tags        = {}
      # (6 unchanged attributes hidden)



      # (8 unchanged blocks hidden)
  }

I am seeing this issue with aws_wafv2_web_acl with AWS provider 3.74.0 and terraform 1.1.5.

Also affects aws_kms_key resources - the policy statement principals are changing order:

  ~ resource "aws_kms_key" "key" {
        id                       = ...
      ~ policy                   = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws:iam::XXXXXXXXXXXX:root",
                                "arn:aws:iam::YYYYYYYYYYYY:root",
                              + "arn:aws:iam::XXXXXXXXXXXX:root",
                            ]
                        }
                        # (4 unchanged elements hidden)
                    },
...

Yes, we got hit with exactly this 😕

@nantiferov I ended up in this thread for issues exactly like what you have described, but upgrading to TF v1.2.1 and AWS provider 4.16.0 has removed the output which I was also getting with every plan previously.

Hi everyone, is there some news about this ? Thanks in advance ?