talos: Inconsistent first boot with Proxmox

Bug Report

Description

I am building a Talos cluster (v1.8.3) on Proxmox (v8.2.2) with Terraform (v1.9.5). Sometimes, on first boot after provisioning, the machines boot to the expected Talos UI and others times they boot and throw the error error running phase 6 in initialize sequence: task 1/1: failed, unexpected EOF followed by failed to revert bootloader: unexpected EOF, then the system reboots and loops this error. I have completely destroyed the terraform managed infrastructure (i.e. Proxmox vm, disks etc) and retried with mixed results. Sometimes they boot as expected and other times they throw the previously mentioned error.

I’ve read in the following issues that I may need certain image extensions, so I have included i915-ucode, intel-ucode, and mei in my image factory generated Talos image, same error and boot loop behavior. I’ve confirmed the id generated with talos_image_factory_schematic matches the id I received from the Image Factory if performed manually.

Observations

I am leveraging a nocloud image so that I may set an IP and Gateway IP address. When running a ping to one of the machines, on first boot the machine does not respond. On second boot right before it enters the rebooting countdown the machine does respond. Only mentioning this as I am unsure if my problem is related to something with nocloud / cloudinit.

IaC

data "talos_image_factory_extensions_versions" "this" {
  talos_version = "v1.8.3"
  filters = {
    names = [
      "i915-ucode",       # see https://github.com/siderolabs/talos/issues/9776
      "intel-ucode",      # see https://github.com/siderolabs/talos/issues/9776
      "iscsi-tools",      # for longhorn
      "mei",              # see https://github.com/siderolabs/talos/issues/9776
      "qemu-guest-agent", # for proxmox
    ]
  }
}

resource "talos_image_factory_schematic" "this" {
  schematic = yamlencode(
    {
      customization = {
        systemExtensions = {
          officialExtensions = data.talos_image_factory_extensions_versions.this.extensions_info.*.name
        }
      }
    }
  )
}

resource "proxmox_virtual_environment_download_file" "talos_image" {
  content_type = "iso"
  datastore_id = "isos"
  node_name    = "pve"
  url          = format("https://factory.talos.dev/image/%s/%s/nocloud-amd64.iso", talos_image_factory_schematic.this.id, data.talos_image_factory_extensions_versions.this.talos_version)
}

resource "proxmox_virtual_environment_vm" "talos_cp" {
  name            = "talos-0"
  node_name       = "pve"
  scsi_hardware   = "virtio-scsi-pci"
  stop_on_destroy = true
  tags            = ["terraform", "linux", "kubernetes", "talos"]
  vm_id           = 210

  agent {
    enabled = true
  }
  startup {
    order      = "3"
    up_delay   = "60"
    down_delay = "60"
  }
  cpu {
    cores = 2
    type  = "x86-64-v2-AES"
  }
  memory {
    dedicated = 2048
    floating  = 2048 # set equal to dedicated to enable ballooning
  }
  disk {
    datastore_id = "truenas-vmware-datastore-0"
    file_id      = proxmox_virtual_environment_download_file.talos_image.id
    interface    = "scsi0"
    size         = 10
  }
  initialization {
    datastore_id = "truenas-vmware-datastore-0"
    dns {
      servers = ["10.0.3.24"]
    }
    ip_config {
      ipv4 {
        address = "10.0.3.151/24"
        gateway = "10.0.3.1"
      }
    }
  }
  network_device {
    bridge   = "vmbr1"
    firewall = true
    model    = "virtio"
    vlan_id  = "3"
  }
  operating_system {
    type = "l26"
  }
}

Logs

image

image

Environment

  • Talos version: v1.8.3
  • Kubernetes version: na
  • Platform: Proxmox v8.2.2 (Intel® Xeon® CPU D-1541)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Would this be with the latest v1.9.0 release?

yes!

Once it’s there, it will fail to boot, so you won’t see it on a successful run.

Anyways worth checking with Talos 1.9 which has a fix