talos: 1.9.1 does not detect NVME disks anymore (Intel N5000)

Bug Report

I upgraded my node from 1.8.4 to 1.9.1 it boots fine but does not find the nvme device anymore:

➜  k8s-prod git:(main) ✗ talosctl get disks -n 10.0.20.81 -i
NODE   NAMESPACE   TYPE   ID      VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID   MODEL   SERIAL
       runtime     Disk   loop0   1         164 kB   true                                                
       runtime     Disk   loop1   1         4.1 kB   true                                                
       runtime     Disk   loop2   1         7.5 MB   true                                                
       runtime     Disk   loop3   1         57 kB    true                                                
       runtime     Disk   loop4   1         2.7 MB   true                                                
       runtime     Disk   loop5   1         2.1 MB   true                                                
       runtime     Disk   loop6   1         4.1 kB   true                                                
       runtime     Disk   loop7   1         127 kB   true                                                
       runtime     Disk   loop8   1         74 MB    true    

The same command on another node with the same hardware still running 1.8.4 shows that:

➜  k8s-prod git:(main) ✗ talosctl get disks -n 10.0.20.82   
WARNING: 10.0.20.82: server version 1.8.3 is older than client version 1.9.1
NODE         NAMESPACE   TYPE   ID        VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID                                   MODEL                   SERIAL
10.0.20.82   runtime     Disk   dm-0      1         88 MB    false                                                                                               
10.0.20.82   runtime     Disk   dm-1      1         499 GB   false                                                                                               
10.0.20.82   runtime     Disk   loop0     1         156 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop1     1         4.1 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop2     1         6.7 MB   true                                                                                                
10.0.20.82   runtime     Disk   loop3     1         57 kB    true                                                                                                
10.0.20.82   runtime     Disk   loop4     1         2.6 MB   true                                                                                                
10.0.20.82   runtime     Disk   loop5     1         4.1 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop6     1         4.1 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop7     1         119 kB   true                                                                                                
10.0.20.82   runtime     Disk   loop8     1         75 MB    true                                                                                                
10.0.20.82   runtime     Disk   nvme0n1   1         500 GB   false       nvme                     eui.002538dc31a109cd                   Samsung SSD 980 500GB   XXXXXXXXXXXXXXXX

These are my extensions:

          officialExtensions:
            - siderolabs/gasket-driver
            - siderolabs/intel-ucode
            - siderolabs/i915 (-ucode in 1.8.x)
            - siderolabs/iscsi-tools
            - siderolabs/util-linux-tools
            - siderolabs/zfs

Description

Unfortunatly most of the other commands like dmesg or support do not work in maintenance mode. What else can I provide? I could provide you with the support bundle if I reboot the 10.0.20.81 node with the old 1.8.4 version…

Logs

This is the lspci output running talos 1.8.4:

root@master1:~# lspci  -v
00:00.0 Host bridge: Intel Corporation Gemini Lake Host Bridge (rev 03)
        Flags: bus master, fast devsel, latency 0
lspci: Unable to load libkmod resources: error -2

00:00.3 System peripheral: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: fast devsel, IRQ 23
        Memory at a1530000 (64-bit, non-prefetchable) [disabled] [size=4K]
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [a0] Vendor Specific Information: Len=14 <?>
        Capabilities: [dc] Power Management version 2
        Capabilities: [f0] PCI Advanced Features

00:02.0 VGA compatible controller: Intel Corporation GeminiLake [UHD Graphics 605] (rev 03) (prog-if 00 [VGA controller])
        DeviceName:  Onboard IGD
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 144
        Memory at a0000000 (64-bit, non-prefetchable) [size=16M]
        Memory at 90000000 (64-bit, prefetchable) [size=256M]
        I/O ports at f000 [size=64]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Len=0c <?>
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [100] Process Address Space ID (PASID)
        Capabilities: [200] Address Translation Service (ATS)
        Capabilities: [300] Page Request Interface (PRI)
        Kernel driver in use: i915

00:0e.0 Audio device: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 25
        Memory at a1510000 (64-bit, non-prefetchable) [size=16K]
        Memory at a1000000 (64-bit, non-prefetchable) [size=1M]
        Capabilities: [50] Power Management version 3
        Capabilities: [80] Vendor Specific Information: Len=14 <?>
        Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00

00:0f.0 Communication controller: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 255
        Memory at a152f000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [a4] Vendor Specific Information: Len=14 <?>

00:12.0 SATA controller: Intel Corporation Celeron/Pentium Silver Processor SATA Controller (rev 03) (prog-if 01 [AHCI 1.0])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 125
        Memory at a1514000 (32-bit, non-prefetchable) [size=8K]
        Memory at a152e000 (32-bit, non-prefetchable) [size=256]
        I/O ports at f090 [size=8]
        I/O ports at f080 [size=4]
        I/O ports at f060 [size=32]
        Memory at a152d000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Kernel driver in use: ahci

00:13.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 120
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: [disabled] [16-bit]
        Memory behind bridge: a1400000-a14fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [140] Access Control Services
        Capabilities: [150] Null
        Capabilities: [200] Null
        Kernel driver in use: pcieport

00:13.2 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 121
        Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
        I/O behind bridge: [disabled] [16-bit]
        Memory behind bridge: a1100000-a12fffff [size=2M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [140] Access Control Services
        Capabilities: [150] Null
        Capabilities: [200] Null
        Kernel driver in use: pcieport

00:13.3 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 122
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
        I/O behind bridge: e000-efff [size=4K] [16-bit]
        Memory behind bridge: a1300000-a13fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: [disabled] [64-bit]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Null
        Capabilities: [140] Access Control Services
        Capabilities: [150] Null
        Capabilities: [200] Null
        Kernel driver in use: pcieport

00:15.0 USB controller: Intel Corporation Celeron/Pentium Silver Processor USB 3.0 xHCI Controller (rev 03) (prog-if 30 [XHCI])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: medium devsel, IRQ 123
        Memory at a1500000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [70] Power Management version 2
        Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
        Capabilities: [90] Vendor Specific Information: Len=14 <?>
        Kernel driver in use: xhci_hcd

00:16.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 0 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 27
        Memory at a152c000 (64-bit, non-prefetchable) [size=4K]
        Memory at a152b000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:16.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 1 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 28
        Memory at a152a000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1529000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:16.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 2 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 29
        Memory at a1528000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1527000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:16.3 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 3 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 30
        Memory at a1526000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1525000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 4 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 31
        Memory at a1524000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1523000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 5 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 32
        Memory at a1522000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1521000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 6 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 33
        Memory at a1520000 (64-bit, non-prefetchable) [size=4K]
        Memory at a151f000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:17.3 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 7 (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 34
        Memory at a151e000 (64-bit, non-prefetchable) [size=4K]
        Memory at a151d000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:19.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
        DeviceName:  Onboard LAN
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 35
        Memory at a151c000 (64-bit, non-prefetchable) [size=4K]
        Memory at a151b000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:19.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at a151a000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1519000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:19.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 37
        Memory at a1518000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1517000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] Vendor Specific Information: Len=14 <?>

00:1f.0 ISA bridge: Intel Corporation Celeron/Pentium Silver Processor LPC Controller (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, medium devsel, latency 0

00:1f.1 SMBus: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: medium devsel, IRQ 20
        Memory at a1516000 (64-bit, non-prefetchable) [size=256]
        I/O ports at f040 [size=32]
        Kernel driver in use: i801_smbus

03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd Device a801
        Flags: bus master, fast devsel, latency 0, IRQ 22
        Memory at a1400000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=13 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158] Power Budgeting <?>
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Latency Tolerance Reporting
        Capabilities: [190] L1 PM Substates
        Kernel driver in use: nvme

05:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU (prog-if ff)
        Subsystem: Global Unichip Corp. Coral Edge TPU
        Flags: bus master, fast devsel, latency 0, IRQ 20
        Memory at a1200000 (64-bit, prefetchable) [size=16K]
        Memory at a1100000 (64-bit, prefetchable) [size=1M]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=128 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
        Capabilities: [108] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [200] Advanced Error Reporting
        Kernel driver in use: apex

06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
        Flags: bus master, fast devsel, latency 0, IRQ 21
        I/O ports at e000 [size=256]
        Memory at a1304000 (64-bit, non-prefetchable) [size=4K]
        Memory at a1300000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
        Capabilities: [170] Latency Tolerance Reporting
        Capabilities: [178] L1 PM Substates
        Kernel driver in use: r8169

Environment

  • Talos version: [talosctl version --nodes <problematic nodes>] 1.9.1
  • Kubernetes version: [kubectl version --short] 1.30.8
  • Platform: bare metal

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

Can you please do talosctl get pcidevices on 1.8/1.9? Trying to see the diff