terraform-provider-aws: EIP scope change incorrectly detected

Community Note

  • Please vote on this issue by adding a ๐Ÿ‘ reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave โ€œ+1โ€ or โ€œme tooโ€ comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform version
Terraform v0.11.7
+ provider.aws v1.28.0

Affected Resource(s)

-/+ aws_eip.sandbox_infrastructure_zappi_it_us_east_1b (new resource required)
      id:                                                     "eipalloc-8b296e83" => <computed> (forces new resource)
      allocation_id:                                          "" => <computed>
      association_id:                                         "" => <computed>
      domain:                                                 "standard" => <computed>
      instance:                                               "i-ac10f68c" => <computed>
      network_interface:                                      "" => <computed>
      private_ip:                                             "" => <computed>
      public_ip:                                              "23.21.180.181" => <computed>
      tags.%:                                                 "0" => "4"
      tags.Environment:                                       "" => "sandbox"
      tags.Infrastructure:                                    "" => "kubernetes"
      tags.Name:                                              "" => "us-east-1b.sandbox.infrastructure.zappi.it"
      tags.Role:                                              "" => "operations"
      vpc:                                                    "false" => "true" (forces new resource)

Terraform Configuration Files

resource "aws_eip" "sandbox_infrastructure_zappi_it_us_east_1b" {
  vpc = true

  depends_on = [
    "aws_internet_gateway.sandbox_infrastructure_zappi_it"
  ]

  tags = {
    Name            = "us-east-1b.sandbox.infrastructure.zappi.it"
    Environment     = "sandbox"
    Infrastructure  = "kubernetes"
    Role            = "operations"
  }
}

Debug Output

Expected Behavior

Terraform should not incorrectly detect that the VPC scope has changed when it has not, as per the screenshot below:

image

Actual Behavior

Terraform incorrectly detects that the VPC scope has changed from false to true.

The example above was not the first incident, but just a more concrete example. Below is the first case we identified:

-/+ aws_eip.us-east-1b-uat-kubernetes-zappi-it (new resource required)
      id:                                                 "eipalloc-855eacb4" => <computed> (forces new resource)
      allocation_id:                                      "" => <computed>
      association_id:                                     "" => <computed>
      domain:                                             "standard" => <computed>
      instance:                                           "i-ac10f68c" => <computed>
      network_interface:                                  "" => <computed>
      private_ip:                                         "" => <computed>
      public_ip:                                          "23.21.180.181" => <computed>
      tags.%:                                             "0" => "3"
      tags.KubernetesCluster:                             "" => "uat.kubernetes.zappi.it"
      tags.Name:                                          "" => "us-east-1b.uat.kubernetes.zappi.it"
      tags.kubernetes.io/cluster/uat.kubernetes.zappi.it: "" => "owned"
      vpc:                                                "false" => "true" (forces new resource)

-/+ aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it (new resource required)
      id:                                                 "nat-050bee284fbf24f25" => <computed> (forces new resource)
      allocation_id:                                      "eipalloc-855eacb4" => "${aws_eip.us-east-1b-uat-kubernetes-zappi-it.id}" (forces new resource)
      network_interface_id:                               "eni-8ff31264" => <computed>
      private_ip:                                         "10.81.2.116" => <computed>
      public_ip:                                          "34.193.141.134" => <computed>
      subnet_id:                                          "subnet-68d74754" => "subnet-68d74754"
      tags.%:                                             "3" => "3"
      tags.KubernetesCluster:                             "uat.kubernetes.zappi.it" => "uat.kubernetes.zappi.it"
      tags.Name:                                          "us-east-1b.uat.kubernetes.zappi.it" => "us-east-1b.uat.kubernetes.zappi.it"
      tags.kubernetes.io/cluster/uat.kubernetes.zappi.it: "owned" => "owned"

  ~ aws_route.private-us-east-1b-0-0-0-0--0
      nat_gateway_id:                                     "nat-050bee284fbf24f25" => "${aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it.id}"

Error during apply:

Applying uat terraform plan...
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Destroying... (ID: nat-050bee284fbf24f25)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 10s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 20s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 30s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 40s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 50s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Destruction complete after 50s
aws_eip.us-east-1b-uat-kubernetes-zappi-it: Destroying... (ID: eipalloc-855eacb4)

Error: Error applying plan:

1 error(s) occurred:

* aws_eip.us-east-1b-uat-kubernetes-zappi-it (destroy): 1 error(s) occurred:

* aws_eip.us-east-1b-uat-kubernetes-zappi-it: InvalidParameterValue: Invalid value 'eipalloc-855eacb4' for PublicIp. Not a valid IPv4 address.
    status code: 400, request id: 9a2c55d3-aa28-4503-b4f0-51e607f5dd07

This behaviour started today at ~08:45 AM UTC for 1 of 3 availability zones for one environment initially.

However, in subsequent terraform plan executions a few minutes later, all of our environments are now affected.

Steps to Reproduce

We use the following during our CI runs:

  1. terraform init
  2. terraform plan -no-color --out="${environment}.plan" | tee "${environment}.report"
  3. terraform apply "${environment}.plan"

Donโ€™t have an isolated reproducible case yet, but will provide if necessary.

Important Factoids

Running in VPC. Terraform is run in CI.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 10
  • Comments: 16 (4 by maintainers)

Most upvoted comments

@rv-aburdine certainly wouldnโ€™t hurt!

Terraform Plan (DEBUG)

Request

2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 10:44:11 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeAddresses Details:
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: POST / HTTP/1.1
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Host: ec2.us-east-1.amazonaws.com
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: User-Agent: aws-sdk-go/1.14.26 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Length: 76
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Authorization: <REDACTED>
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: X-Amz-Date: 20180725T164411Z
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Accept-Encoding: gzip
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Action=DescribeAddresses&AllocationId.1=eipalloc-de3a31d6&Version=2016-11-15

Note the allocation ID above: eipalloc-de3a31d6

Response

2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 10:44:12 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeAddresses Details:
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: HTTP/1.1 200 OK
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Connection: close
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Transfer-Encoding: chunked
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Type: text/xml;charset=UTF-8
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Date: Wed, 25 Jul 2018 16:44:12 GMT
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Server: AmazonEC2
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Vary: Accept-Encoding
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: -----------------------------------------------------
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 10:44:12 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: <DescribeAddressesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     <requestId>b2b333cd-e839-47b7-9fba-a27f20d61c2d</requestId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     <addressesSet>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>23.21.219.184</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId/>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>54.225.190.133</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId/>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>107.22.209.166</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId>i-ff4e8d8e</instanceId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>174.129.10.84</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId>i-6761af1a</instanceId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>34.192.126.190</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <allocationId>eipalloc-de3a31d6</allocationId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>vpc</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <associationId>eipassoc-14144abf</associationId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <networkInterfaceId>eni-b7c2a786</networkInterfaceId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <networkInterfaceOwnerId>670359441688</networkInterfaceOwnerId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <privateIpAddress>10.105.0.76</privateIpAddress>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     </addressesSet>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: </DescribeAddressesResponse>

Note the allocation ID eipalloc-de3a31d6 is returned, but itโ€™s the last element in the response XML. Note the first element in the response XML is IP 23.21.219.184

Plan Output

-/+ module.vpc.aws_eip.nat2 (new resource required)
      id:                                        "eipalloc-de3a31d6" => <computed> (forces new resource)
      allocation_id:                             "" => <computed>
      association_id:                            "" => <computed>
      domain:                                    "standard" => <computed>
      instance:                                  "" => <computed>
      network_interface:                         "" => <computed>
      private_ip:                                "" => <computed>
      public_ip:                                 "23.21.219.184" => <computed>
      vpc:                                       "false" => "true" (forces new resource)

Note the public_ip property is showing a current value of 23.21.219.184 โ€“ the first element in the ec2/DescribeAddresses response.

AWS CLI

aws ec2 describe-addresses --allocation-ids eipalloc-de3a31d6
{
    "Addresses": [
        {
            "InstanceId": "",
            "PublicIp": "23.21.219.184",
            "Domain": "standard"
        },
        {
            "InstanceId": "",
            "PublicIp": "54.225.190.133",
            "Domain": "standard"
        },
        {
            "InstanceId": "i-ff4e8d8e",
            "PublicIp": "107.22.209.166",
            "Domain": "standard"
        },
        {
            "InstanceId": "i-6761af1a",
            "PublicIp": "174.129.10.84",
            "Domain": "standard"
        },
        {
            "PublicIp": "34.192.126.190",
            "AllocationId": "eipalloc-de3a31d6",
            "AssociationId": "eipassoc-14144abf",
            "Domain": "vpc",
            "NetworkInterfaceId": "eni-b7c2a786",
            "NetworkInterfaceOwnerId": "670359441688",
            "PrivateIpAddress": "10.105.0.76"
        }
    ]
}

After executing the AWS CLI command several times, I was able to get one correct response:

aws ec2 describe-addresses --allocation-ids eipalloc-de3a31d6
{
    "Addresses": [
        {
            "PublicIp": "34.192.126.190",
            "AllocationId": "eipalloc-de3a31d6",
            "AssociationId": "eipassoc-14144abf",
            "Domain": "vpc",
            "NetworkInterfaceId": "eni-b7c2a786",
            "NetworkInterfaceOwnerId": "670359441688",
            "PrivateIpAddress": "10.105.0.76"
        }
    ]
}

This seems to indicate there is a rollout in progress.