terraform-provider-aws: [Bug]: IPAM allocation fails with "InvalidIpamPoolAllocationId"

Hello Kevin, thanks for putting the effort in for the sample code! In the provider we have mechanisms for retries and waiting (retries and waiters), and our PR guidelines suggest that we follow any exiting patterns in the resource being modified.

I have added the mechanisms for retry and waiting to account for eventually consistency of the read operation, and I’ve added additional acceptance tests to verify cross region pool CIDR allocation.

I assure you this is being worked on. The provider team does releases each Thursday.

I can successfully reproduce in an AccTests. Working on a fix.

We’re seeing similar issues with that resource as well (using 4.50.0). IPAM pool isn’t shared with RAM in our case, all operations happen in the same AWS account.

First attempt

plan

# aws_vpc_ipam_pool_cidr_allocation.workload[0] will be created
+ resource "aws_vpc_ipam_pool_cidr_allocation" "workload" {
  + cidr                    = (known after apply)
  + description             = "some desc"
  + id                      = (known after apply)
  + ipam_pool_allocation_id = (known after apply)
  + ipam_pool_id            = "ipam-pool-xxxxxxxxxx"
  + netmask_length          = 25
  + resource_id             = (known after apply)
  + resource_owner          = (known after apply)
  + resource_type           = (known after apply)
}

apply

apply fails with the following error message, however if we check the AWS Console the allocation is well created in IPAM service.

Error: reading IPAM Pool CIDR Allocation (ipam-pool-alloc-xxxxxxxxxxx_ipam-pool-xxxxxxxxxx): couldn't find resource

Second attempt

plan

Resource is shown as tainted.

# aws_vpc_ipam_pool_cidr_allocation.workload[0] is tainted, so must be replaced
-/+ resource "aws_vpc_ipam_pool_cidr_allocation" "workload" {
  + cidr                    = (known after apply)
  ~ id                      = "ipam-pool-alloc-xxxxxxxxxxx_ipam-pool-xxxxxxxxxx" -> (known after apply)
  + ipam_pool_allocation_id = (known after apply)
  + resource_id             = (known after apply)
  + resource_owner          = (known after apply)
  + resource_type           = (known after apply)
    # (3 unchanged attributes hidden)
}

apply

Because the resource is tainted, it is being deleted, but that fails as well.

Error: deleting IPAM Pool CIDR Allocation (ipam-pool-alloc-xxxxxx_ipam-pool-xxxxx): InvalidParameterValue: The CIDR specified :  is not in proper format.

(^ not a typo in error message, a value is missing)

EDIT: We’ve opened a case with AWS support in the meantime, as we believe this is likely to be an issue with AWS IPAM service API rather than the provider. We were able to replicate the issue with AWS CLI as well.

Hi @Tailzip - Can you tell me the CLI steps to reproduce. When I do the following aws ec2 get-ipam-pool-allocations --ipam-pool-id POOL-ID --ipam-pool-allocation-id ALLOCATION-ID I consistently get correct result.

AWS will not support unless we show them that their CLI also fails

A quick clarification, AWS Enterprise Support does offer Third-Party Product support, including open source software such as Terraform. I agree that having a reproducible case in a script using the AWS CLI is certainly helpful, though not required.

AWS works with Hashicorp and the open source community to evaluate and prioritize issues as per the Terraform AWS Provider FAQ.

I can successfully reproduce in an AccTests. Working on a fix.

I also just started on this issue and added this code block to ipam_pool_cidr_allocation.go

	// Handle eventual consitency of the API and therefor retry the read
	return resource.Retry(time.Minute, func() *resource.RetryError {
		err = resourceIPAMPoolCIDRAllocationRead(d, meta)

		if err != nil {
			if tfresource.NotFound(err) {
				return resource.RetryableError(fmt.Errorf("IPAM Pool CIDR Allocation (%s) not yet ready", d.Id()))
			} else {
				return resource.NonRetryableError(err)
			}
		}

		return nil
	})

We need this change urgently. Do you work on this within the next few days or should I open a PR? If the latter, could you share your test code?

@AdamTylerLynch I will do that. I just thought it was worth mentioning that using import was not a workaround for us.

Hi @Tailzip - Can you tell me the CLI steps to reproduce. When I do the following aws ec2 get-ipam-pool-allocations --ipam-pool-id POOL-ID --ipam-pool-allocation-id ALLOCATION-ID I consistently get correct result. AWS will not support unless we show them that their CLI also fails

I’ve been running the following script, and issue happens randomly after a couple runs

script.sh

#!/bin/bash

set -e

export AWS_REGION=eu-central-1
export AWS_DEFAULT_REGION=eu-central-1
export AWS_DEFAULT_OUTPUT=json

IPAM_POOL_ID="ipam-pool-xxxxxxxxxxxxxxxxx"
ALLOCATION_ID="$(aws ec2 allocate-ipam-pool-cidr --ipam-pool-id "$IPAM_POOL_ID" --netmask-length 25 --description 'troubleshoot' | jq -c -r '.IpamPoolAllocation.IpamPoolAllocationId')"

aws ec2 get-ipam-pool-allocations \
    --ipam-pool-id "$IPAM_POOL_ID" \
    --ipam-pool-allocation-id "$ALLOCATION_ID" \
    --no-cli-pager

Interesting. For me the script always runs through without any issues and the creation through terraform still throws the error InvalidIpamPoolAllocationId.NotFound. Exact same IAM-Role used.

Mili Durasovic mili.durasovic@mercedes-benz.com, Mercedes-Benz Tech Innovation GmbH Provider Information

Hi @Tailzip - Can you tell me the CLI steps to reproduce. When I do the following aws ec2 get-ipam-pool-allocations --ipam-pool-id POOL-ID --ipam-pool-allocation-id ALLOCATION-ID I consistently get correct result.

AWS will not support unless we show them that their CLI also fails

I’ve been running the following script, and issue happens randomly after a couple runs

#!/bin/bash

set -e

export AWS_REGION=eu-central-1
export AWS_DEFAULT_REGION=eu-central-1
export AWS_DEFAULT_OUTPUT=json

IPAM_POOL_ID="ipam-pool-xxxxxxxxxxxxxxxxx"
ALLOCATION_ID="$(aws ec2 allocate-ipam-pool-cidr --ipam-pool-id "$IPAM_POOL_ID" --netmask-length 25 --description 'troubleshoot' | jq -c -r '.IpamPoolAllocation.IpamPoolAllocationId')"

aws ec2 get-ipam-pool-allocations \
    --ipam-pool-id "$IPAM_POOL_ID" \
    --ipam-pool-allocation-id "$ALLOCATION_ID" \
    --no-cli-pager

We have the exact same problem as @Tailzip. We are referencing the cidr in a local. It seems that the local is being evaluated way too early. The terraform resource might indicate that the allocation is done, but it seems like it’s still ongoing asynchronously in AWS.

We tried to remove the tainted resource and then import the resource, but that doesn’t seem to work. Afterwards it showed us a completely new resource being created. Applying that will result in the mentioned error again (“couldn’t find resource”). The import statement looks a little bit weird as well. The text says that the allocation id is used for the import, but the example only shows the resource.

EDIT: We verified the problem with 4.19.0, 4.46.0 and 4.48.0. It seems that it started last week as sporadic behavior, but now this is a constant behavior.