talos: Bare-Metal: Multiple Kernel panics

Bug Report

I am getting multiple Kernel panics in my first foray into Bare-metal Talos nodes.

After investigation (below) it seems that

  • Broadcom 9500-8i HBAs doesnt seem to be supported
  • Another seems to be due to Cilium (see answers below)
  • Yet another seems to be due to the Talos Kernel missing a commit that patches CVE-2024-39277 .

Description

Server stuck in a bootloop and I see a lot of repeating stacktraces that seem to indicate some HBA driver or config missing. Not sure how to fix it.

Logs

03/09/2024 22:57:39
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.6.43-talos #1
03/09/2024 22:57:39
Hardware name: Supermicro Super Server/H13SAE-MF, BIOS 2.0a 06/05/2024
03/09/2024 22:57:39
[drm] Detected VRAM RAM=512M, BAR=512M
03/09/2024 22:57:39
mpt3sas_cm0: vphy entry: 0000000032f3ac7f, port id: 0, phy:8 is added to port's vphys_list
03/09/2024 22:57:39
Call Trace:
03/09/2024 22:57:39
<IRQ>
03/09/2024 22:57:39
dump_stack_lvl+0x47/0x60
03/09/2024 22:57:39
[drm] RAM width 128bits DDR5
03/09/2024 22:57:39
mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500062b20ec64880), phys(11)
03/09/2024 22:57:39
__ubsan_handle_out_of_bounds+0xc7/0x100
03/09/2024 22:57:39
[drm] amdgpu: 512M of VRAM memory ready
03/09/2024 22:57:39
mpt3sas_scsih_event_callback+0xca7/0xd60 [mpt3sas]
03/09/2024 22:57:39
[drm] amdgpu: 63967M of GTT memory ready.
03/09/2024 22:57:39
mpt3sas_cm0: port enable: SUCCESS
03/09/2024 22:57:39
? rebalance_domains+0x189/0x450
03/09/2024 22:57:39
[drm] GART: num cpu pages 262144, num gpu pages 262144
03/09/2024 22:57:39
_base_process_reply_queue+0x538/0x1330 [mpt3sas]
03/09/2024 22:57:39
[drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
03/09/2024 22:57:39
? srso_alias_return_thunk+0x5/0xfbef5
03/09/2024 22:57:39
[drm] Loading DMUB firmware via PSP: version=0x05000F00
03/09/2024 22:57:39
? _nohz_idle_balance.isra.0+0x100/0x340
03/09/2024 22:57:39
_base_interrupt+0x43/0x60 [mpt3sas]
03/09/2024 22:57:39
[drm] use_doorbell being set to: [true]
03/09/2024 22:57:39
__handle_irq_event_percpu+0x4f/0x190
03/09/2024 22:57:39
[drm] Found VCN firmware Version ENC: 1.30 DEC: 3 VEP: 0 Revision: 8
03/09/2024 22:57:39
handle_irq_event+0x31/0x80
03/09/2024 22:57:39
handle_edge_irq+0x9b/0x290
03/09/2024 22:57:39
amdgpu 0000:11:00.0: amdgpu: Will use PSP to load VCN firmware SUBSYSTEM=pci DEVICE=+pci:0000:11:00.0
03/09/2024 22:57:39
__common_interrupt+0x4e/0xe0
03/09/2024 22:57:39
[drm] reserve 0xa00000 from 0xf41e000000 for PSP TMR
03/09/2024 22:57:39
common_interrupt+0x81/0xa0
03/09/2024 22:57:39
amdgpu 0000:11:00.0: amdgpu: RAS: optional ras ta ucode is not available SUBSYSTEM=pci DEVICE=+pci:0000:11:00.0
03/09/2024 22:57:39
</IRQ>
03/09/2024 22:57:39
amdgpu 0000:11:00.0: amdgpu: RAP: optional rap ta ucode is not available SUBSYSTEM=pci DEVICE=+pci:0000:11:00.0
03/09/2024 22:57:39
<TASK>
03/09/2024 22:57:39
asm_common_interrupt+0x26/0x40
03/09/2024 22:57:39
RIP: 0010:cpuidle_enter_state+0xd9/0x6a0

Environment

  • Talos version: 1.7.6
  • Kubernetes version: 1.30.3
  • Platform: bare-metal

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments