terraform-provider-aws: Terraform cannot destroy VPC Links to API Gateway because of a dependency issue

Hi,

I also stumbled across this error. I tried the suggestion to use the lifecycle hook “create_before_destory”, but unfortunately without success. After some research, I came across a note in the AWS doc (https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-known-issues.html) which says that the VPC link should not be referenced directly, but instead a stage variable should be used as a reference. I set this up and can now reliably delete VPC links.

For an API Resource or Method entity with a private integration, you should delete it after removing any hard-coded reference of a VpcLink. Otherwise, you have a dangling integration and receive an error stating that the VPC link is still in use even when the Resource or Method entity is deleted. This behavior does not apply when the private integration references VpcLink through a stage variable.

resource "aws_api_gateway_deployment" "example" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  stage_name  = var.env

  variables = {
    "vpc_link_id" = aws_api_gateway_vpc_link.example.id
  }

  depends_on = [
    aws_api_gateway_vpc_link.example,
    // other resources
  ]

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_vpc_link" "example" {
  name        = "${var.project}-example-${var.env}"
  target_arns = [var.loadbalancer_arn]
}

resource "aws_api_gateway_integration" "example" {
  integration_http_method = "ANY"
  type                    = "HTTP_PROXY"
  connection_type         = "VPC_LINK"
  connection_id           = "$${stageVariables.vpc_link_id}" //weird, but correct syntax!
  // ...
}

It appears the problem is that API Gateways are “compiled” and deployed instances of what the console is representing. This is a problem that I experienced in the past because you would change the API Gateway using the console and the API Gateway wouldn’t respond to the updated changes until you “deploy api” and this takes your definition in the console, builds a distribution and pushes it to cloudfront.

I think this is the real problem here. I’ve been doing some testing and found that the crossed dependencies make nearly imposible to change parameters on resources that trigger a recreation of that resource.

While trying to find a way to past the VPC_LINK problem I tried to change the integration_type that is refering that VPC_LINK to MOCK. This change is triggering a recreation of the integration object but is failing because it’s a required dependency for the route that is using it.

Error: error deleting API Gateway v2 VPC Link (l4er5c): BadRequestException: Cannot delete vpc link. Vpc link 'l4er5c', is referenced in [ANY:qtfgl2b] in format of [Method:Resource].

Error: error deleting API Gateway v2 integration: ConflictException: Cannot delete Integration because it is referenced by the following Routes with Ids: [qtfgl2b]

I’m trying to play with the depends_on meta parameter from terraform but without luck so far 😦

EDIT: To make things even worse, you cannot define two different vpc-links to the same subnets even with different security-groups. This is an AWS API limitation:

aws apigatewayv2 create-vpc-link --name mgm-test --subnet-ids subnet-aaaaaaa subnet-bbbbbbb subnet-ccccccc  --security-group-ids sg-aaaaaaaaaaaaaa sg-bbbbbbbbbbbbb

An error occurred (InternalServerException) when calling the CreateVpcLink operation (reached max retries: 2): Internal server error

As the VPC-LINK is creating ENIs attached to those subnets seems that it can only attach one per subnet.

So you cannot create a new one with the configuration you need and then switch the integration to the new one so you don’t have downtime.

It appears the problem is that API Gateways are “compiled” and deployed instances of what the console is representing. This is a problem that I experienced in the past because you would change the API Gateway using the console and the API Gateway wouldn’t respond to the updated changes until you “deploy api” and this takes your definition in the console, builds a distribution and pushes it to cloudfront.

If you look in the cloudfront console, there is no distribution since API Gateway cloudfront distributions are hidden and only accessible through API Gateways Console interface.

So the problem is, that terraform wants to delete/recreate/change a VPC Link, so it tries to do that, but there is a connection to this “object” in the cloudfront distribution and this blocks modification of the VPC Link.

So even if you delete the resources in API Gateways interface, but don’t deploy it, you still can’t modify this VPC Link, because the link is being referenced in the compiled, distributed version that is installed on cloudfront.

If you remove the methods and integrations from your API Gateway and remove the reference to the VPC link, then deploy api, THEN you can delete the vpc link.

Terraform apparently doesn’t know how to do this, but there are no mechanisms that I’m aware of other than scripting to make a sequence of actions that will do the following

1. Remove the integration from API Gateway
2. Deploy the api (breaking the online distribution temporarily)
3. Delete/Recreate/Edit the VPC Link according to the terraform definition
4. Create any methods/integrations necessary, relinking the VPC Link to the API Gateway
5. Deploying the new api

This sequence of actions in theory would work. However I can’t see any way to do them using terraform primatives. Anybody has any ideas how you’d do this in pure terraform.tf files?

Figured out a solution! The aws_api_gateway_deployment resource documentation has a note talking about adding a create_before_destroy lifecycle block. This allows the VPC Link to connect to the new API Gateway Deployment first before destroying the old API Gateway Deployment, so at all times the VPC Link is attached and prevents the downtime.

lifecycle { create_before_destroy = true }

Oh you’re new to this? First time? (Movie quote bonus! +1)

Don’t worry, the problem will disappear and then come back, at least that’s what it does with me 😃 On 17. Jul 2020, 13:53 +0200, Steven Massaro notifications@github.com, wrote:

I’m wondering if you could cleverly use null_resources, targeting a script that always deploys the api, and a clever use of target and depends_on to trigger a series of two deployments. First, when the link changes therefore severing the link to the rest api and the second execution to deploy the changed resources with the new link. Do you understand what I mean? … On 14. Jul 2020, 17:34 +0200, Steven Massaro @.***>, wrote: Unfortunately no ideas for you Chris. This problem does effectively make it impossible to automate the changes to API Gateway if it contains VPC links. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Unfortunately I don’t Chris. I’m quite new to Terraform. Nevertheless, this is (magically, frustratingly) no longer an issue for me. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.