terraform-provider-aws: [Bug]: "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element

Terraform Core Version

1.3.5

AWS Provider Version

4.45.0

Affected Resource(s)

aws_wafv2_web_acl

Expected Behavior

We should now use rule_action_override

           dynamic rule_action_override {
              for_each = rule.value.excluded_rules
              content {
                action_to_use {
                  count {}
                }
                name = rule_action_override.value
              }
            }

instead of deprecated excluded_rule

            dynamic "excluded_rule" {
              for_each = rule.value.excluded_rules
              content {
                name = excluded_rule.value
              }
            }

Actual Behavior

When using dynamic rule_action_override block, the webacl gets created or updated as expected. However, subsequent updates are impossible : Error: Provider produced inconsistent final plan

Reverting to excluded_rule allows new updates to the webacl.

Relevant Error/Panic Output Snippet

Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes
  
Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.test_alb_webacl to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"block":cty.List(cty.Object(map[string]cty.Type{"custom_response":cty.List(cty.Object(map[string]cty.Type{"custom_response_body_key":cty.String,
"response_code":cty.Number, "response_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"captcha":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"count":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))}))})), "name":cty.StringVal("AWSManagedRulesCommonRuleSet"),
"override_action":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"count":cty.ListValEmpty(cty.EmptyObject),
"none":cty.ListVal([]cty.Value{cty.EmptyObjectVal})})}), "priority":cty.NumberIntVal(2100),
"rule_label":cty.SetValEmpty(cty.Object(map[string]cty.Type{"name":cty.String})),
"statement":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"and_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"and_statement":cty.List(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"and_statement":cty.List(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"byte_match_statement":cty.List(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
"body":cty.List(cty.EmptyObject),
"cookies":cty.List(cty.Object(map[string]cty.Type{"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"excluded_cookies":cty.List(cty.String), "included_cookies":cty.List(cty.String)})), "match_scope":cty.String,
"oversize_handling":cty.String})),
"headers":cty.List(cty.Object(map[string]cty.Type{"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"excluded_headers":cty.List(cty.String), "included_headers":cty.List(cty.String)})), "match_scope":cty.String,
"oversize_handling":cty.String})),
"json_body":cty.List(cty.Object(map[string]cty.Type{"invalid_fallback_behavior":cty.String,
"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"included_paths":cty.List(cty.String)})), "match_scope":cty.String, "oversize_handling":cty.String})),
"method":cty.List(cty.EmptyObject), "query_string":cty.List(cty.EmptyObject),
"single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"uri_path":cty.List(cty.EmptyObject)})), "positional_constraint":cty.String, "search_string":cty.String,
"text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number, "type":cty.String}))})),
"geo_match_statement":cty.List(cty.Object(map[string]cty.Type{"country_codes":cty.List(cty.String),
"forwarded_ip_config":cty.List(cty.Object(map[string]cty.Type{"fallback_behavior":cty.String,
"header_name":cty.String}))})), "ip_set_reference_statement":cty.List(cty.Object(map[string]cty.Type{"arn":cty.String,
"ip_set_forwarded_ip_config":cty.List(cty.Object(map[string]cty.Type{"fallback_behavior":cty.String,
"header_name":cty.String, "position":cty.String}))})),
"label_match_statement":cty.List(cty.Object(map[string]cty.Type{"key":cty.String, "scope":cty.String})),
"regex_match_statement":cty.List(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
"body":cty.List(cty.EmptyObject),

[...]

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

resource "aws_wafv2_web_acl" "test_alb_webacl" {
  name        = "${var.test_vpc_name}"
  description = "WebACL for ${var.test_vpc_name}"
  scope       = "REGIONAL"

  default_action {
    block {}
  }

[...]

  # Managed rules
  dynamic "rule" {
    for_each = var.test_rule_groups
      content {
        name     = rule.key
        priority = rule.value.priority

        override_action {
          none {}
        }

        statement {
          managed_rule_group_statement {
            name        = rule.key
            vendor_name = rule.value.vendor_name

            dynamic rule_action_override {
               for_each = rule.value.excluded_rules
               content {
                 action_to_use {
                   count {}
                 }
    
                 name = rule_action_override.value
               }
             } 

          }
        }
        
      }
  }

[...]


}

Steps to Reproduce

1- Create or update webacl to use rule_action_override (and apply) 2- Modify something in the code that will require a webacl update (and apply) --> this will produce an inconsistant final plan 3- Revert to excluded_rule (and apply) --> works fine and further webacl updates are possible

Debug Output

No response

Panic Output

No response

Important Factoids

Tested on Ubuntu 22.04

References

No response

Would you like to implement a fix?

None

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 35
Comments: 35 (12 by maintainers)

Most upvoted comments

Thank you @bclodius for the work around!! I can confirm that changing action_to_use.count to include the custom_requrest_handling.insert-header block was able to get me past the issue on AWS v4.53.0. Seems like a fairly high severity bug since it prevents a successful terraform apply. I hope it gets fixed soon.

+13

timharsch on Feb 9, 2023

Issue debugging

@ewbankkit I think I found something leading to part of the root cause here based on some investigation I did on the plan json that can reproduce the issue and a “workaround”. Will dig more when I get a chance but this seems to be a workaround in my case.

This schema here is an optional list: https://github.com/hashicorp/terraform-provider-aws/blob/f7cf1351f83ed0124725a65382a00b91931ff1c0/internal/service/wafv2/schemas.go#L591

When I add or modify any aws_wafv2_web_acl.rule attribute or add a new aws_wafv2_web_acl.rule it creates the following discrepancy for all rules for aws_wafv2_web_acl.rule.statement.managed_rule_group_statement.rule_action_override.action_to_user.*.custom_request_handling

In my plan file’s json the resource_changes section shows this…

# Before section...
"action_to_use": [
                            {
                              "allow": [],
                              "block": [],
                              "captcha": [],
                              "count": [
                                {
                                  "custom_request_handling": null
                                }
                              ]
                            }
                          ]

# After section...
"action_to_use": [
                            {
                              "allow": [],
                              "block": [],
                              "captcha": [],
                              "count": [
                                {
                                  "custom_request_handling": []
                                }
                              ]
                            }
                          ]

Workaround

If I ensure there is something populated in custom_request_handling then this error is fixed for my use case.

# BEFORE AND CAUSING ISSUES
dynamic "rule_action_override" {
              for_each = managed_rule_group_statement.value.count_rule_action_overrides
              content {
                action_to_use {
                  count {
                  }
                }
                name = rule_action_override.key
              }
            }

# AFTER
dynamic "rule_action_override" {
              for_each = managed_rule_group_statement.value.count_rule_action_overrides
              content {
                action_to_use {
                  count {
                  custom_request_handling {
                      insert_header {
                        name  = "TFWorkaround"
                        value = "WAF"
                      }
                    }
                  }
                }
                name = rule_action_override.key
              }
            }

Once I make these changes the before and after plans still show changes to all the rules but don’t crash on apply.

bclodius on Jan 20, 2023

I am experiencing the same issue, but not with the action_to_use override construct, but with a regular allow or count within a dynamic rule creation.

Using the same method described above, the workaround is successful here too.

  dynamic "rule" {
    for_each = var.custom_rules

    content {
      name     = rule.key
      priority = try(rule.value["priority"], "100")

      # What rule action do we want to apply
      action {
        dynamic "allow" {
          for_each = rule.value["action"] == "allow" ? [true] : []
          content {
            # --------------------------------------------------------------------
            # Adding a custom header as a workaround for a BUG in the AWS provider
            # See https://github.com/hashicorp/terraform-provider-aws/issues/28191
            # --------------------------------------------------------------------
            custom_request_handling {
              insert_header {
                name  = "TFWorkaround"
                value = "WAF"
              }
            }
          }
        }
  :
  :
  :

Hasgaroth on May 2, 2023

Hey all, I have come across a similar issue this week too. I’m aiming to look deeper into the issue and implement a fix hopefully. If anyone is interested in pairing with me on this issue, that would be great!

Update:

I have done some digging and it could be any of these 2 issues which have stood out. The first issue could be with the way the custom request handling is handled during the creation and update of web ACLs.

In the func resourceWebACLCreate in web_acl.go that is here - it seems that we need to handle this (custom_request_handling) in the webACL.
Another potential issue is that the schema mentioned by @bclodius in his workaround; yes, the customRequestHandlingSchema is optional, however, the insert_header schema is set to required and perhaps it could also be this. The children of insert_header (name & value) are also set to required. Perhaps making the insert_header to optional could be a fix ?

If Go experts could take a look at this and perhaps comment or suggest something, it would be awesome!

moabukar on Apr 2, 2023

Also present in v4.60.0.

asininemonkey on Mar 27, 2023

Also see this error. Do we know when a fix will be in place?

olearylynda on Mar 22, 2023

As a workaround I have been tainting my web acl when rules are impacting changes to webacl.

I keep the same webacl name so that my WAF logs do not get impacted.

Edit: See next comment below for a workaround that works for me

bclodius on Feb 1, 2023

Having the same issue, Terraform 1.3.3 and AWS provider 4.63.0

smartpierre on Apr 18, 2023

@amitsamal94 the workaround just ensures that the plan doesn’t end up in a situation that causes the error. The bug gets triggered when there’s NOTHING overridden in the override. In my case I had to force an extra header in the rule override config. This extra header doesn’t impact my application.

bclodius on Mar 15, 2023