kubitect: Bridge mode fails

Hi,

thank you so much for this project, it is really a life saver.

Recently I have been trying to create a bridged network and assign static IPs to all machines but I keep failing with message:

Error: couldn't retrieve IP address of domain id: 7c5e009c-443d-4ba4-a61a-0d9bbf3f61d3. Please check following:
1) is the domain running proplerly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 timeout while waiting for state to become 'all-addresses-obtained' (last state: 'waiting-addresses', timeout: 5m0s)

  with module.worker_module["1"].libvirt_domain.vm_domain,
  on modules/vm/vm.tf line 58, in resource "libvirt_domain" "vm_domain":
  58: resource "libvirt_domain" "vm_domain" {

Basically it keeps waiting for ips even though they are assigned statically and times out after 5 minutes:

2022-03-01T10:34:39.842Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] waiting for network address for iface=52:54:00:6C:3C:01
2022-03-01T10:34:39.842Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] qemu-agent used to query interface info
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] Interfaces info obtained with libvirt API:
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: ([]libvirt.DomainInterface) <nil>
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14:
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] ifaces with addresses: []
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] 52:54:00:6C:3C:01 doesn't have IP address(es) yet...
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] IP address not found for iface=52:54:00:6C:3C:01: will try in a while
2022-03-01T10:34:39.843Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [TRACE] Waiting 10s before next try
2022-03-01T10:34:39.880Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] waiting for network address for iface=52:54:00:6C:3C:02
2022-03-01T10:34:39.880Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] qemu-agent used to query interface info
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] Interfaces info obtained with libvirt API:
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: ([]libvirt.DomainInterface) <nil>
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14:
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] ifaces with addresses: []
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] 52:54:00:6C:3C:02 doesn't have IP address(es) yet...
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [DEBUG] IP address not found for iface=52:54:00:6C:3C:02: will try in a while
2022-03-01T10:34:39.881Z [DEBUG] provider.terraform-provider-libvirt_v0.6.14: 2022/03/01 10:34:39 [TRACE] Waiting 10s before next try

Does anyone know why this is happening?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (8 by maintainers)

Most upvoted comments

In general, I don’t think proxy is a problem because if your bridge interface gets its own IP address, so should the virtual machines. This is just a guess though, as I’ve no idea how your network is implemented.

I’m still not able to reproduce the issue other than with incorrect values, so it seems to me that it needs further investigation on your end. If the NAT mode is sufficient for your needs, that should do for now.

Please let me know if you have any more questions or information about this problem.

MusicDin on Mar 11, 2022

I was able to recreate this issue.

For example, I have my network configured as follows: CIDR (for LAN network): 10.10.0.0/20 GW (router’s IP): 10.10.0.1

If I enter the following values when creating the cluster, the cluster gets successfully created:

# terraform.tfvars

network_mode = "bridge"
network_bridge = "br0"
network_cidr = "10.10.0.0/20"
network_gateway = "10.10.0.1" # In this case, GW can be omitted
...
master_nodes = [
  {
    id  = 1
    ip  = "10.10.6.5"
  }
]
...
worker_nodes = [
  {
    id  = 1
    ip  = "10.10.6.6"
  }
]

If I enter the wrong GW IP, the addresses are not retrieved and I get the same error message as you. The same thing happens if the wrong network CIDR is specified. For example, if I enter network_cidr = "10.10.0.0/22", I again get the same error as you.

Can you verify that you enterd the correct CIDR and GW?

CIDR should be your LAN network with the appropriate subnet mask network bits,
GW should point to the router of the specified network,
the static addresses assigned to the VMs should be within the specified network.

MusicDin on Mar 3, 2022