talos: Inconsistent first boot with Proxmox
Bug Report
Description
I am building a Talos cluster (v1.8.3) on Proxmox (v8.2.2) with Terraform (v1.9.5). Sometimes, on first boot after provisioning, the machines boot to the expected Talos UI and others times they boot and throw the error error running phase 6 in initialize sequence: task 1/1: failed, unexpected EOF followed by failed to revert bootloader: unexpected EOF, then the system reboots and loops this error. I have completely destroyed the terraform managed infrastructure (i.e. Proxmox vm, disks etc) and retried with mixed results. Sometimes they boot as expected and other times they throw the previously mentioned error.
I’ve read in the following issues that I may need certain image extensions, so I have included i915-ucode, intel-ucode, and mei in my image factory generated Talos image, same error and boot loop behavior. I’ve confirmed the id generated with talos_image_factory_schematic matches the id I received from the Image Factory if performed manually.
- Talos 1.8.0+ initial boot fails in phase meta (6/12) #9776
- Node stuck in boot loop on Talos v1.8.0 upgrade #9369
- Can not boot Talos 1.8.* #9529
Observations
I am leveraging a nocloud image so that I may set an IP and Gateway IP address. When running a ping to one of the machines, on first boot the machine does not respond. On second boot right before it enters the rebooting countdown the machine does respond. Only mentioning this as I am unsure if my problem is related to something with nocloud / cloudinit.
IaC
data "talos_image_factory_extensions_versions" "this" {
talos_version = "v1.8.3"
filters = {
names = [
"i915-ucode", # see https://github.com/siderolabs/talos/issues/9776
"intel-ucode", # see https://github.com/siderolabs/talos/issues/9776
"iscsi-tools", # for longhorn
"mei", # see https://github.com/siderolabs/talos/issues/9776
"qemu-guest-agent", # for proxmox
]
}
}
resource "talos_image_factory_schematic" "this" {
schematic = yamlencode(
{
customization = {
systemExtensions = {
officialExtensions = data.talos_image_factory_extensions_versions.this.extensions_info.*.name
}
}
}
)
}
resource "proxmox_virtual_environment_download_file" "talos_image" {
content_type = "iso"
datastore_id = "isos"
node_name = "pve"
url = format("https://factory.talos.dev/image/%s/%s/nocloud-amd64.iso", talos_image_factory_schematic.this.id, data.talos_image_factory_extensions_versions.this.talos_version)
}
resource "proxmox_virtual_environment_vm" "talos_cp" {
name = "talos-0"
node_name = "pve"
scsi_hardware = "virtio-scsi-pci"
stop_on_destroy = true
tags = ["terraform", "linux", "kubernetes", "talos"]
vm_id = 210
agent {
enabled = true
}
startup {
order = "3"
up_delay = "60"
down_delay = "60"
}
cpu {
cores = 2
type = "x86-64-v2-AES"
}
memory {
dedicated = 2048
floating = 2048 # set equal to dedicated to enable ballooning
}
disk {
datastore_id = "truenas-vmware-datastore-0"
file_id = proxmox_virtual_environment_download_file.talos_image.id
interface = "scsi0"
size = 10
}
initialization {
datastore_id = "truenas-vmware-datastore-0"
dns {
servers = ["10.0.3.24"]
}
ip_config {
ipv4 {
address = "10.0.3.151/24"
gateway = "10.0.3.1"
}
}
}
network_device {
bridge = "vmbr1"
firewall = true
model = "virtio"
vlan_id = "3"
}
operating_system {
type = "l26"
}
}
Logs
Environment
- Talos version: v1.8.3
- Kubernetes version: na
- Platform: Proxmox v8.2.2 (Intel® Xeon® CPU D-1541)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (12 by maintainers)
Commits related to this issue
- chore: add more debugging logs for META and volumes Doesn't fix anything, but the hope is that it would help with #9852 and #9786. Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com> — committed to smira/talos by smira 2 years ago
- chore: add more debugging logs for META and volumes Doesn't fix anything, but the hope is that it would help with #9852 and #9786. Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com> (cher... — committed to smira/talos by smira 2 years ago
- fix: many small fixes, mostly around overwrites 1. Push GPT discovery last in the chain - if the GPT image is overridden with a smaller filesystem image, it might detect GPT before actual files... — committed to smira/go-blockdevice by smira 2 years ago
- fix: many small fixes, mostly around overwrites 1. Push GPT discovery last in the chain - if the GPT image is overridden with a smaller filesystem image, it might detect GPT before actual files... — committed to smira/go-blockdevice by smira 2 years ago
- fix: many small fixes, mostly around overwrites 1. Push GPT discovery last in the chain - if the GPT image is overridden with a smaller filesystem image, it might detect GPT before actual files... — committed to smira/go-blockdevice by smira 2 years ago
yes!
Once it’s there, it will fail to boot, so you won’t see it on a successful run.
Anyways worth checking with Talos 1.9 which has a fix