terraform-provider-aws: API Gateway v2 integration & route: ConflictException due to concurrent modification

Getting error when creating multiple aws_apigatewayv2_integration and aws_apigatewayv2_route resources

error creating API Gateway v2 integration: ConflictException: Unable to complete operation due to concurrent modification. Please try again later.

error creating API Gateway v2 route: ConflictException: Unable to complete operation due to concurrent modification. Please try again later

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.6 AWS provider 3.24.0

Affected Resource(s)

aws_apigatewayv2_integration
aws_apigatewayv2_route

Terraform Configuration Files

resource "aws_apigatewayv2_integration" "lambda" {
  for_each               = local.lambda_resources
  api_id                 = aws_apigatewayv2_api.this.id
  credentials_arn        = aws_iam_role.api.arn
  integration_type       = "AWS_PROXY"
  connection_type        = "INTERNET"
  integration_method     = "POST"
  integration_uri        = each.value
  payload_format_version = "2.0"
}

resource "aws_apigatewayv2_route" "lambda" {
  for_each  = local.lambda_resources
  api_id    = aws_apigatewayv2_api.this.id
  route_key = "${each.value.method} ${each.value.endpoint}"
  target    = "integrations/${aws_apigatewayv2_integration.lambda[each.key].id}"
}

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 38
Comments: 19 (4 by maintainers)

Most upvoted comments

I am experiencing a similar issue when trying to create aws_apigatewayv2_route. In my case, we have a module that allows a map of routes to be passed in, which are then created using a for_each. We are seeing about a 40% failure rate with the error below. Sometimes it works flawlessly, other times it fails. It is very intermittent which also makes it hard to reproduce consistently.

error creating API Gateway v2 route: ConflictException: Unable to complete operation due to concurrent modification. Please try again later

It appears a similar issue was fixed for apigatewayv1 by adding retry logic here: https://github.com/hashicorp/terraform-provider-aws/blob/5e6e4dfee3c1e97336cb996fffe4fbf4a5270134/aws/config.go#L679-L686

Could we potentially add that same fix for apigatewayv2?

Burnett2k on Jul 20, 2021

@maryelizbeth 👋 sorry to tag you directly, by a long shot is anything you or anyone in your team can help getting this issue back on the radar ?

Nowadays a lot of folks are using either v1 or v2 of the API GW and i’m very much surprised that up until now this issue hasn’t been a priority.

Would appreciate if the priority gets bumped (i’m not there yet to contribute , otherwise i would have done it … already spent 2 weeks fighting with this issue)

DanyC97 on Nov 9, 2021

@ewbankkit tried to lower parallelism to 5, but it didn’t help. And got error for creating aws_apigatewayv2_route

error creating API Gateway v2 route: ConflictException: Unable to complete operation due to concurrent modification. Please try again later

But the error is not consistent

ghost on Mar 12, 2021

I think the related commit that was implemented for API Gateway v1 (and REST API) is this one https://github.com/hashicorp/terraform-provider-aws/commit/d906813c60bd8f5b664849498932f4473c4105fe it has been shuffled around since and the piece of code is now located here https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/conns/config.go#L261

I’m not familiar with AWS Go SDK internals nor Terraform handlers but would it be possible to use something very similar for API Gateway v2 by importing github.com/aws/aws-sdk-go/service/apigatewayv2 in internal/conns/config.go and then adding the following right next to https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/conns/config.go#L261

client.apigatewayv2.Handlers.Retry.PushBack(func(r *request.Request) {
	if tfawserr.ErrMessageContains(r.Error, apigatewayv2.ErrCodeConflictException, "try again later") {
		r.Retryable = aws.Bool(true)
	}
})

it seems the apigatewayv2 package in AWS Go SDK implements the exact same Exception https://github.com/aws/aws-sdk-go/blob/main/service/apigatewayv2/errors.go#L28 vs https://github.com/aws/aws-sdk-go/blob/main/service/apigateway/errors.go#L23

remil1000 on Feb 28, 2023

Seeing the same error when trying to create a simple serverless function with a custom authorizer. This is a bad bug.

Resource handler returned
message: “Unable to complete operation due to concurrent
modification. Please try
again later. (Service:
ApiGateway, Status Code:
409, Request ID: fa501120-9c 3f-4efa-90c0-355bc14c00cb,
Extended Request ID: null)”
(RequestToken: 93f797df-
fe7d-5e0d-9d16-287af48122e7, HandlerErrorCode:
AlreadyExists) The following resource(s)
failed to create:
[CustomAuthorizer]. Rollback requested by user.

si3mshady on Dec 7, 2021

I’m also seeing this, with aws 3.57. I have less than a dozen routes+integrations being created from a map by a module, and i get one to three ConflictExceptions like this for the integrations at least half the time i try to apply. Adding -parallelism=1 seems to work around it 100% of the time, though i wouldn’t say my testing has been exhaustive yet.

I think the v1 fix mentioned in the previous comment is related to the nearly identical issue #16439.

okdana on Sep 17, 2021