terraform-provider-aws: RouteTable association reports empty result - and apply fails, but routetable is properly associated in AWS

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform 1.0.1 linux amd64

• Installed hashicorp/aws v3.64.2 (signed by HashiCorp)
• Installed hashicorp/random v3.1.0 (signed by HashiCorp)
• Installed hashicorp/time v0.7.2 (signed by HashiCorp)

Affected Resource(s)

• aws_route_table_association

Terraform Configuration Files

Associated routing resources (sans VPC, subnets, internet-gateway, and nat)

//-----------------------------------------------------------
// public route table and internet gateway route
//-----------------------------------------------------------
resource "aws_route_table" "publicRouteTable" {
  vpc_id = aws_vpc.mainVPC.id
  tags = {
    Name    = "${var.environmentName} Public route table"
  }
}

//note that tags are not allowed on a route
resource "aws_route" "publicInternetGatewayRoute" {
  route_table_id = aws_route_table.publicRouteTable.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = aws_internet_gateway.mainIG.id
  timeouts {
    create = "6m"
  }
}

//-----------------------------------------------------------
// private route table and nat gateway route
//-----------------------------------------------------------
resource "aws_route_table" "privateRouteTable" {
  vpc_id = aws_vpc.mainVPC.id
  tags = {
    Name    = "${var.environmentName} Private route table"
  }
}

//note that tags are not allowed on a route
resource "aws_route" "privateNatGatewayRoute" {
  route_table_id          = aws_route_table.privateRouteTable.id
  destination_cidr_block  = "0.0.0.0/0"
  nat_gateway_id          = aws_nat_gateway.natGateway.id
  timeouts {
    create = "6m"
  }
}


//-----------------------------------------------------------
// route table associations
//-----------------------------------------------------------
resource "time_sleep" "waitRouteTableElements" {
  depends_on = [
    aws_route_table.publicRouteTable,
    aws_route.publicInternetGatewayRoute,
    aws_route_table.privateRouteTable,
    aws_route.privateNatGatewayRoute,
    aws_subnet.publicSubNet1,
    aws_subnet.publicSubNet2,
    aws_subnet.privateSubNet1,
    aws_subnet.privateSubNet2
  ]
  create_duration = "6m"
}

resource "aws_route_table_association" "public1" {
  depends_on     = [time_sleep.waitRouteTableElements]
  subnet_id      = aws_subnet.publicSubNet1.id
  route_table_id = aws_route_table.publicRouteTable.id
}

resource "aws_route_table_association" "public2" {
  depends_on     = [time_sleep.waitRouteTableElements]
  subnet_id      = aws_subnet.publicSubNet2.id
  route_table_id = aws_route_table.publicRouteTable.id
}

resource "aws_route_table_association" "private1" {
  depends_on     = [time_sleep.waitRouteTableElements]
  subnet_id      = aws_subnet.privateSubNet1.id
  route_table_id = aws_route_table.privateRouteTable.id
}

resource "aws_route_table_association" "private2" {
  depends_on     = [time_sleep.waitRouteTableElements]
  subnet_id      = aws_subnet.privateSubNet2.id
  route_table_id = aws_route_table.privateRouteTable.id
}

Debug Output

Panic Output

Expected Behavior

Identified that AWS reported the association was complete, and not failed the apply.

Actual Behavior

Looks like it might not be waiting long enough. Loops waiting for DescribeRouteTables to indicate associated:

<AssociateRouteTableResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
    <requestId>aea99d68-b36e-4fe7-b267-d91e827d285a</requestId>
    <associationId>rtbassoc-07d32038d3161359d</associationId>
    <associationState>
        <state>associated</state>
    </associationState>
</AssociateRouteTableResponse>: timestamp=2021-11-08T11:30:07.613-0500
2021-11-08T11:30:07.613-0500 [INFO]  provider.terraform-provider-aws_v3.64.2_x5: 2021/11/08 11:30:07 [DEBUG] Waiting for Route Table Association (rtbassoc-07d32038d3161359d) creation: timestamp=2021-11-08T11:30:07.613-0500
2021-11-08T11:30:07.613-0500 [INFO]  provider.terraform-provider-aws_v3.64.2_x5: 2021/11/08 11:30:07 [DEBUG] Waiting for state to become: [associated]: timestamp=2021-11-08T11:30:07.613-0500
2021-11-08T11:30:07.614-0500 [INFO]  provider.terraform-provider-aws_v3.64.2_x5: 2021/11/08 11:30:07 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeRouteTables Details:

-----------------------------------------------------: timestamp=2021-11-08T11:30:07.760-0500
2021-11-08T11:30:07.789-0500 [INFO]  provider.terraform-provider-aws_v3.64.2_x5: 2021/11/08 11:30:07 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeRouteTables Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK

Gets a payload response that indicates its associated:

<DescribeRouteTablesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
    <requestId>d8bec8c3-4a8a-45ba-8420-cdf7d1354e31</requestId>
    <routeTableSet>
        <item>
            <routeTableId>rtb-0a8ba52dfbbdc7717</routeTableId>
            <vpcId>vpc-06f9c23d15c361252</vpcId>
            <ownerId>#######</ownerId>
            <routeSet>
                <item>
                    <destinationCidrBlock>10.100.0.0/16</destinationCidrBlock>
                    <gatewayId>local</gatewayId>
                    <state>active</state>
                    <origin>CreateRouteTable</origin>
                </item>
                <item>
                    <destinationCidrBlock>0.0.0.0/0</destinationCidrBlock>
                    <gatewayId>igw-0c8ab62f7a92f1c37</gatewayId>
                    <state>active</state>
                    <origin>CreateRoute</origin>
                </item>
            </routeSet>
            <associationSet>
                <item>
                    <routeTableAssociationId>rtbassoc-099a83c99f258c36f</routeTableAssociationId>
                    <routeTableId>rtb-0a8ba52dfbbdc7717</routeTableId>
                    <subnetId>subnet-0fc53a3b7ebe19d64</subnetId>
                    <main>false</main>
                    <associationState>
                        <state>associated</state>
                    </associationState>
                </item>
                <item>
                    <routeTableAssociationId>rtbassoc-0dcdc8c9a5e92d331</routeTableAssociationId>
                    <routeTableId>rtb-0a8ba52dfbbdc7717</routeTableId>
                    <subnetId>subnet-00621c15caee268dc</subnetId>
                    <main>false</main>
                    <associationState>
                        <state>associated</state>
                    </associationState>
                </item>
            </associationSet>
            <propagatingVgwSet/>
            <tagSet>
                <item>
                    <key>User</key>
                    <value>####</value>
                </item>
                <item>
                    <key>Name</key>
                    <value>#### Public route table</value>
                </item>
                <item>
                    <key>Domain</key>
                    <value>####</value>
                </item>
                <item>
                    <key>Billing</key>
                    <value>####</value>
                </item>
            </tagSet>
        </item>
    </routeTableSet>
</DescribeRouteTablesResponse>: timestamp=2021-11-08T11:30:07.789-0500
2021-11-08T11:30:07.790-0500 [INFO]  provider.terraform-provider-aws_v3.64.2_x5: 2021/11/08 11:30:07 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeRouteTables Details:

but then it fails with : empty result

module.network.aws_route_table_association.private1: Creating...
module.network.aws_route_table_association.private2: Creation complete after 1s [id=rtbassoc-05ad72d7a0531ee28]
module.network.aws_route_table_association.public1: Creation complete after 1s [id=rtbassoc-0dcdc8c9a5e92d331]
module.network.aws_route_table_association.public2: Creation complete after 1s [id=rtbassoc-099a83c99f258c36f]

Error: error reading Route Table Association (rtbassoc-07d32038d3161359d): empty result

  with module.network.aws_route_table_association.private1,
  on network/baseNetwork.tf line 184, in resource "aws_route_table_association" "private1":
 184: resource "aws_route_table_association" "private1" {

Steps to Reproduce

This does not happen all the time. Only occasionally, always on a fresh deploy (all we are doing right now).

1. terraform apply

Important Factoids

There doesn’t appear to be an adjustable timeout for aws_route_table_association

References

• #0000