terraform-provider-proxmox: Failed to create qemu VMs: 599 Too Many Redirections
tldr; When trying to create 3 qemu VMs on the same host (slowly), the provider is unable to handle the transient 599 error raised by the proxmox API, causing an error during apply.
proposed solution to handle & retry on 599 errors
Terraform code:
variable "template_id" {
description = "The VM ID of the template node to deploy from"
default = "kube-ubuntu-2004-20230529-220351"
type = string
}
variable "controlplane_memory" {
description = "MB of RAM to allocate for control planes"
type = number
default = 2048
}
variable "controlplane_cpu" {
description = "CPU cores to allocate for control planes"
type = number
default = 2
}
locals {
timestamp = replace(replace(timestamp(), "-", ""), ":", "")
target_node = "pve-1"
controlplanes = {
"1" = {
target_node = "pve-1"
hastate = "" # no HA
hagroup = ""
id = 9010
},
"2" = {
target_node = "pve-1"
hastate = "" # no HA
hagroup = ""
id = 9011
},
"3" = {
target_node = "pve-1"
hastate = ""
hagroup = ""
id = 9012
}
}
}
resource "proxmox_vm_qemu" "controlplane" {
for_each = local.controlplanes
vmid = each.value.id
name = "k8s-master-${local.timestamp}-${each.key}"
target_node = each.value.target_node
clone = var.template_id
onboot = true # Start nodes on boot
agent = 1 # Qemu guest agent
hastate = each.value.hastate
hagroup = each.value.hagroup
memory = var.controlplane_memory
cores = var.controlplane_cpu
scsihw = "virtio-scsi-pci"
cicustom = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
vga {
type = "virtio"
}
}
Plan:
~ cd ./infra/controlplane/terraform && bash run.sh
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of telmate/proxmox from the dependency lock file
- Using previously-installed telmate/proxmox v2.9.14
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
proxmox_vm_qemu.controlplane["3"]: Refreshing state... [id=pve-1/qemu/9012]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
+ create
Terraform will perform the following actions:
# proxmox_vm_qemu.controlplane["1"] will be created
+ resource "proxmox_vm_qemu" "controlplane" {
+ additional_wait = 5
+ agent = 1
+ automatic_reboot = true
+ balloon = 0
+ bios = "seabios"
+ boot = (known after apply)
+ bootdisk = (known after apply)
+ cicustom = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
+ clone = "kube-ubuntu-2004-20230530-013945"
+ clone_wait = 10
+ cores = 2
+ cpu = "host"
+ default_ipv4_address = (known after apply)
+ define_connection_info = true
+ force_create = false
+ full_clone = true
+ guest_agent_ready_timeout = 100
+ hotplug = "network,disk,usb"
+ id = (known after apply)
+ kvm = true
+ memory = 2048
+ name = (known after apply)
+ nameserver = (known after apply)
+ onboot = true
+ oncreate = true
+ preprovision = true
+ reboot_required = (known after apply)
+ scsihw = "virtio-scsi-pci"
+ searchdomain = (known after apply)
+ sockets = 1
+ ssh_host = (known after apply)
+ ssh_port = (known after apply)
+ tablet = true
+ target_node = "pve-1"
+ unused_disk = (known after apply)
+ vcpus = 0
+ vlan = -1
+ vmid = 9010
+ vga {
+ type = "virtio"
}
}
# proxmox_vm_qemu.controlplane["2"] will be created
+ resource "proxmox_vm_qemu" "controlplane" {
+ additional_wait = 5
+ agent = 1
+ automatic_reboot = true
+ balloon = 0
+ bios = "seabios"
+ boot = (known after apply)
+ bootdisk = (known after apply)
+ cicustom = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
+ clone = "kube-ubuntu-2004-20230530-013945"
+ clone_wait = 10
+ cores = 2
+ cpu = "host"
+ default_ipv4_address = (known after apply)
+ define_connection_info = true
+ force_create = false
+ full_clone = true
+ guest_agent_ready_timeout = 100
+ hotplug = "network,disk,usb"
+ id = (known after apply)
+ kvm = true
+ memory = 2048
+ name = (known after apply)
+ nameserver = (known after apply)
+ onboot = true
+ oncreate = true
+ preprovision = true
+ reboot_required = (known after apply)
+ scsihw = "virtio-scsi-pci"
+ searchdomain = (known after apply)
+ sockets = 1
+ ssh_host = (known after apply)
+ ssh_port = (known after apply)
+ tablet = true
+ target_node = "pve-1"
+ unused_disk = (known after apply)
+ vcpus = 0
+ vlan = -1
+ vmid = 9011
+ vga {
+ type = "virtio"
}
}
# proxmox_vm_qemu.controlplane["3"] will be created
+ resource "proxmox_vm_qemu" "controlplane" {
+ additional_wait = 5
+ agent = 1
+ automatic_reboot = true
+ balloon = 0
+ bios = "seabios"
+ boot = (known after apply)
+ bootdisk = (known after apply)
+ cicustom = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
+ clone = "kube-ubuntu-2004-20230530-013945"
+ clone_wait = 10
+ cores = 2
+ cpu = "host"
+ default_ipv4_address = (known after apply)
+ define_connection_info = true
+ force_create = false
+ full_clone = true
+ guest_agent_ready_timeout = 100
+ hotplug = "network,disk,usb"
+ id = (known after apply)
+ kvm = true
+ memory = 2048
+ name = (known after apply)
+ nameserver = (known after apply)
+ onboot = true
+ oncreate = true
+ preprovision = true
+ reboot_required = (known after apply)
+ scsihw = "virtio-scsi-pci"
+ searchdomain = (known after apply)
+ sockets = 1
+ ssh_host = (known after apply)
+ ssh_port = (known after apply)
+ tablet = true
+ target_node = "pve-1"
+ unused_disk = (known after apply)
+ vcpus = 0
+ vlan = -1
+ vmid = 9012
+ vga {
+ type = "virtio"
}
}
Plan: 3 to add, 0 to change, 0 to destroy.
──────────────────────────────────────
Saved the plan to: plan.out
Apply the plan, note parallelism 1 to try and slow things down.
~ terraform apply -parallelism=1 plan.out
proxmox_vm_qemu.controlplane["3"]: Creating...
proxmox_vm_qemu.controlplane["3"]: Still creating... [10s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [20s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [30s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [40s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [50s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m0s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m10s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m20s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m30s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m40s elapsed]
proxmox_vm_qemu.controlplane["3"]: Creation complete after 1m44s [id=pve-1/qemu/9012]
proxmox_vm_qemu.controlplane["1"]: Creating...
proxmox_vm_qemu.controlplane["1"]: Still creating... [10s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [20s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [30s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [40s elapsed]
virt-customize --truncate /etc/machine-info --truncate /etc/machine-idproxmox_vm_qemu.controlplane["1"]: Still creating... [50s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m0s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m10s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m20s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m30s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m40s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m50s elapsed]
proxmox_vm_qemu.controlplane["1"]: Creation complete after 1m58s [id=pve-1/qemu/9010]
proxmox_vm_qemu.controlplane["2"]: Creating...
proxmox_vm_qemu.controlplane["2"]: Still creating... [10s elapsed]
proxmox_vm_qemu.controlplane["2"]: Still creating... [20s elapsed]
╷
│ Error: 599 Too many redirections
│
│ with proxmox_vm_qemu.controlplane["2"],
│ on main.tf line 1, in resource "proxmox_vm_qemu" "controlplane":
│ 1: resource "proxmox_vm_qemu" "controlplane" {
│
╵
Relevant tflog:
2023-05-30T02:10:54.337-0600 [INFO] Starting apply for proxmox_vm_qemu.controlplane["1"]
2023-05-30T02:10:54.337-0600 [DEBUG] proxmox_vm_qemu.controlplane["1"]: applying the planned Create change
2023-05-30T02:10:54.338-0600 [INFO] provider.terraform-provider-proxmox_v2.9.14: 2023/05/30 02:10:54 [DEBUG] setting computed for "unused_disk" from ComputedKeys: timestamp=2023-05-30T02:10:54.338-0600
2023-05-30T02:10:54.339-0600 [INFO] provider.terraform-provider-proxmox_v2.9.14: 2023/05/30 02:10:54 [DEBUG][QemuVmCreate] checking for duplicate name: k8s-master-20230530T080704Z-1: timestamp=2023-05-30T02:10:54.339-0600
2023-05-30T02:10:54.346-0600 [INFO] provider.terraform-provider-proxmox_v2.9.14: 2023/05/30 02:10:54 [DEBUG][QemuVmCreate] cloning VM: timestamp=2023-05-30T02:10:54.346-0600
proxmox_vm_qemu.controlplane["1"]: Still creating... [10s elapsed]
2023-05-30T02:11:14.243-0600 [ERROR] provider.terraform-provider-proxmox_v2.9.14: Response contains error diagnostic: diagnostic_summary="599 Too many redirections" diagnostic_severity=ERROR tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/telmate/proxmox tf_req_id=22ff0f17-0ebb-8454-5a12-3f6ef9dbcdad tf_resource_type=proxmox_vm_qemu @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= tf_rpc=ApplyResourceChange timestamp=2023-05-30T02:11:14.243-0600
2023-05-30T02:11:14.250-0600 [ERROR] vertex "proxmox_vm_qemu.controlplane[\"1\"]" error: 599 Too many redirections
╷
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 20
We have the same issue on latest master.
I have gotten a response from proxmox staff, should be fixed in new release (5.1.2) of libpve-http-server-perl https://forum.proxmox.com/threads/error-599-too-many-redirections.148788/post-716536 Still waiting for our IT to update the affected cluster, so no feedback from us, yet
same issue on v3.0.1-rc1