terraform-provider-openstack: openstack_networking_port_v2 all_fixed_ips is empty after creation (first run) but filled after rm from state (second run)

Thank you as well for the patience. I’ve updated the docs so hopefully it will be clear for other users. I’ll close the issue.

but likely it’s not tackling the root cause of a deferred / delayed IP address allocation.

I think it will tackle the root cause which based on the neutron people from launchpad is:

This result is something expected if the network where the port is created has no subnets

I’ve actually had deployments where this behavior occurred.

Explaining:

Currently you have this code:

resource "openstack_networking_port_v2" "vpn" {
  name       = "vpn"
  network_id = var.network_id

  admin_state_up     = "true"
  security_group_ids = [openstack_networking_secgroup_v2.vpn.id]
}

This only says to TF that the port resource is dependent to the network resource, nothing about the subnet. Based on the dependency graph TF will parallelize the creation of the subnet AND port => their creation will be triggered “at the same time” meaning you are in a race condition (which you have noticed). Sometimes it will work because internally on neutron level the subnet creation will be before the port creation and thus you will get an IP, other times it will be the opposite and you wont get an IP. If you look at your terraform apply output you probably will see something like:

creating network resource
...
network resource **created** (ID=blah_blah)
creating port resource
creating subnet resource <= (triggered at the same time, you are in a race condition)
...

If you add switch your port resource to:

resource "openstack_networking_port_v2" "vpn" {
  name       = "vpn"
  network_id = var.network_id

  admin_state_up     = "true"
  security_group_ids = [openstack_networking_secgroup_v2.vpn.id]

 fixed_ip {
   subnet_id = openstack_networking_subnet_v2.name-here.id
 }
}

This will make known to TF that the port resources is dependant of the subnet => the TF depedency graph will force the subnet creation to be done before it triggers the port creation. So this should remove the race condition. Your terraform apply logs will look like:

creating network resource
...
network resource **created** (ID=blah_blah)
creating subnet resource
...
subnet resource **created** (ID= bluh bluh)
creating port resource <= triggered after the subnet is created and therefore based on neutron people input your port will get an ip now. there is no race condition
...

depends_on will have the same result but it is a bit more pesky to use when you have for_each etc to create multiple resources.

Given the neutron people input, similar behaviors i’ve noticed and your input (race condition + not using deferred) I am rather certain the above will fix it. I would prefer if you can test the above solution before we consider adding a wait.

@frittentheke are you creating the subnet in the same run? and if so can you add a depends_on on the port resource for the subnet, or alternatively define subnet_id in the port resource => https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/networking_port_v2#subnet_id (inside the fixed_ip block)?

From their response this is expected if the subnet is not created. the above 2 options should force the port creation to happen after the subnet creation.

In the meantime ill try to find some time to check whether we can/should add a “wait” to the port resource.

Is there a way to ask a Neutron dev if this is intended behavior for the API to potentially return the port without fixed_ips populated?

The bug you opened is a way. there is already a response. Otherwise IRC channels are also an option: https://docs.openstack.org/contributors/common/irc.html

@nikParasyr thanks for the fast response!

I did run the scenario you asked for. Full disclosure, there are quite a few resources spawned and the port for the instance running the VPN services is created by a terraform module … but here we go:

1. terraform apply ended with the error I reported about all_fixed_ips being empty.
2. This is the state for the ressource:

terraform state show module.vpn-server.openstack_networking_port_v2.vpn
# module.vpn-server.openstack_networking_port_v2.vpn:
resource "openstack_networking_port_v2" "vpn" {
    admin_state_up         = true
    all_fixed_ips          = []
    all_security_group_ids = [
        "87acb073-5123-4473-b33b-fc78f522c6b8",
    ]
    all_tags               = []
    dns_assignment         = []
    id                     = "9b37978b-ed53-41c2-983f-31570eb88259"
    mac_address            = "fa:16:3e:3a:58:ec"
    name                   = "vpn"
    network_id             = "f946cedc-94d1-4bde-a680-f59d615ad2e3"
    port_security_enabled  = true
    region                 = "fra"
    security_group_ids     = [
        "87acb073-5123-4473-b33b-fc78f522c6b8",
    ]
    tenant_id              = "REDACTED"

    allowed_address_pairs {
        ip_address = "10.3.4.0/24"
    }

    binding {
        vif_details = {}
        vnic_type   = "normal"
    }
}

so all_fixed_ips is empty.

1. Running terraform apply again does not work, it’s ending up with the same error about all_fixed_ips
2. State remains unchanged.

But we dug a little deeper:

1. terraform refresh does NOT update the all_fixed_ips (if called implicitly by the apply or explicitly)
2. terraform apply -target module.vpn-server.openstack_networking_port_v2.vpn does “work”, but finds nothing that needs changing. So also then the field is not populated.
3. Certainly the terraform state rm module.vpn-server.openstack_networking_port_v2.vpn which I did initially of course caused a new port to be created leaving the first one dangling. But then all_fixed_ips was set, so the other resource referring that worked fine.
4. As I said there a quite a few resources in this terraform code, so I believe this is the reason the port resource does not “work” the same way for the first apply doing everything, but for the second attempt with only this port and the static route being changed / created via the API. Read: convergence time.

This is the terraform debug output / openstack API response to the port creation (initial terraform apply):

[...]
2023-08-23T12:06:29.893+0200 [INFO]  provider.terraform-provider-openstack_v1.52.1: 2023/08/23 12:06:29 [DEBUG] OpenStack Request URL: GET https://network.regsion.cloud.example.com/v2.0/ports?id=9b37978b-ed53-41c2-983f-31570eb88259: timestamp=2023-08-23T12:06:
29.893+0200
2023-08-23T12:06:29.893+0200 [INFO]  provider.terraform-provider-openstack_v1.52.1: 2023/08/23 12:06:29 [DEBUG] OpenStack Request Headers:
Accept: application/json
Cache-Control: no-cache
User-Agent: HashiCorp Terraform/1.5.5 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 gophercloud/v1.4.0
X-Auth-Token: ***: timestamp=2023-08-23T12:06:29.893+0200
2023-08-23T12:06:29.983+0200 [INFO]  provider.terraform-provider-openstack_v1.52.1: 2023/08/23 12:06:29 [DEBUG] OpenStack Response Code: 200: timestamp=2023-08-23T12:06:29.983+0200
2023-08-23T12:06:29.983+0200 [INFO]  provider.terraform-provider-openstack_v1.52.1: 2023/08/23 12:06:29 [DEBUG] OpenStack Response Headers:
Content-Type: application/json
Date: Wed, 23 Aug 2023 10:06:29 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000
Vary: Accept-Encoding
Via: 1.1 network.region.cloud.example.com
X-Openstack-Request-Id: req-1b02e0f3-442d-4b85-a9ce-40a765d05fb5: timestamp=2023-08-23T12:06:29.983+0200
2023-08-23T12:06:29.983+0200 [INFO]  provider.terraform-provider-openstack_v1.52.1: 2023/08/23 12:06:29 [DEBUG] OpenStack Response Body: {
  "ports": [
    {
      "admin_state_up": true,
      "allowed_address_pairs": [
        {
          "ip_address": "10.3.4.0/24",
          "mac_address": "fa:16:3e:3a:58:ec"
        }
      ],
      "binding:vnic_type": "normal",
      "created_at": "2023-08-23T10:06:12Z",
      "description": "",
      "device_id": "",
      "device_owner": "",
      "dns_assignment": [],
      "dns_name": "",
      "extra_dhcp_opts": [],
      "fixed_ips": [],
      "id": "9b37978b-ed53-41c2-983f-31570eb88259",
      "mac_address": "fa:16:3e:3a:58:ec",
      "name": "vpn",
      "network_id": "f946cedc-94d1-4bde-a680-f59d615ad2e3",
      "port_security_enabled": true,
      "project_id": "REDACTED",
      "revision_number": 1,
      "security_groups": [
        "87acb073-5123-4473-b33b-fc78f522c6b8"
      ],
      "status": "DOWN",
      "tags": [],
      "tenant_id": "REDACTED",
      "updated_at": "2023-08-23T10:06:12Z"
    }
  ]
}: timestamp=2023-08-23T12:06:29.983+0200
[...]