talos: 1.9.1 does not detect NVME disks anymore (Intel N5000)
Bug Report
I upgraded my node from 1.8.4 to 1.9.1 it boots fine but does not find the nvme device anymore:
➜ k8s-prod git:(main) ✗ talosctl get disks -n 10.0.20.81 -i
NODE NAMESPACE TYPE ID VERSION SIZE READ ONLY TRANSPORT ROTATIONAL WWID MODEL SERIAL
runtime Disk loop0 1 164 kB true
runtime Disk loop1 1 4.1 kB true
runtime Disk loop2 1 7.5 MB true
runtime Disk loop3 1 57 kB true
runtime Disk loop4 1 2.7 MB true
runtime Disk loop5 1 2.1 MB true
runtime Disk loop6 1 4.1 kB true
runtime Disk loop7 1 127 kB true
runtime Disk loop8 1 74 MB true
The same command on another node with the same hardware still running 1.8.4 shows that:
➜ k8s-prod git:(main) ✗ talosctl get disks -n 10.0.20.82
WARNING: 10.0.20.82: server version 1.8.3 is older than client version 1.9.1
NODE NAMESPACE TYPE ID VERSION SIZE READ ONLY TRANSPORT ROTATIONAL WWID MODEL SERIAL
10.0.20.82 runtime Disk dm-0 1 88 MB false
10.0.20.82 runtime Disk dm-1 1 499 GB false
10.0.20.82 runtime Disk loop0 1 156 kB true
10.0.20.82 runtime Disk loop1 1 4.1 kB true
10.0.20.82 runtime Disk loop2 1 6.7 MB true
10.0.20.82 runtime Disk loop3 1 57 kB true
10.0.20.82 runtime Disk loop4 1 2.6 MB true
10.0.20.82 runtime Disk loop5 1 4.1 kB true
10.0.20.82 runtime Disk loop6 1 4.1 kB true
10.0.20.82 runtime Disk loop7 1 119 kB true
10.0.20.82 runtime Disk loop8 1 75 MB true
10.0.20.82 runtime Disk nvme0n1 1 500 GB false nvme eui.002538dc31a109cd Samsung SSD 980 500GB XXXXXXXXXXXXXXXX
These are my extensions:
officialExtensions:
- siderolabs/gasket-driver
- siderolabs/intel-ucode
- siderolabs/i915 (-ucode in 1.8.x)
- siderolabs/iscsi-tools
- siderolabs/util-linux-tools
- siderolabs/zfs
Description
Unfortunatly most of the other commands like dmesg or support do not work in maintenance mode. What else can I provide? I could provide you with the support bundle if I reboot the 10.0.20.81 node with the old 1.8.4 version…
Logs
This is the lspci output running talos 1.8.4:
root@master1:~# lspci -v
00:00.0 Host bridge: Intel Corporation Gemini Lake Host Bridge (rev 03)
Flags: bus master, fast devsel, latency 0
lspci: Unable to load libkmod resources: error -2
00:00.3 System peripheral: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: fast devsel, IRQ 23
Memory at a1530000 (64-bit, non-prefetchable) [disabled] [size=4K]
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [a0] Vendor Specific Information: Len=14 <?>
Capabilities: [dc] Power Management version 2
Capabilities: [f0] PCI Advanced Features
00:02.0 VGA compatible controller: Intel Corporation GeminiLake [UHD Graphics 605] (rev 03) (prog-if 00 [VGA controller])
DeviceName: Onboard IGD
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 144
Memory at a0000000 (64-bit, non-prefetchable) [size=16M]
Memory at 90000000 (64-bit, prefetchable) [size=256M]
I/O ports at f000 [size=64]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 2
Capabilities: [100] Process Address Space ID (PASID)
Capabilities: [200] Address Translation Service (ATS)
Capabilities: [300] Page Request Interface (PRI)
Kernel driver in use: i915
00:0e.0 Audio device: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 25
Memory at a1510000 (64-bit, non-prefetchable) [size=16K]
Memory at a1000000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [50] Power Management version 3
Capabilities: [80] Vendor Specific Information: Len=14 <?>
Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
00:0f.0 Communication controller: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 255
Memory at a152f000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [a4] Vendor Specific Information: Len=14 <?>
00:12.0 SATA controller: Intel Corporation Celeron/Pentium Silver Processor SATA Controller (rev 03) (prog-if 01 [AHCI 1.0])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 125
Memory at a1514000 (32-bit, non-prefetchable) [size=8K]
Memory at a152e000 (32-bit, non-prefetchable) [size=256]
I/O ports at f090 [size=8]
I/O ports at f080 [size=4]
I/O ports at f060 [size=32]
Memory at a152d000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0
Kernel driver in use: ahci
00:13.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 120
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: [disabled] [16-bit]
Memory behind bridge: a1400000-a14fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Capabilities: [a0] Power Management version 3
Capabilities: [100] Null
Capabilities: [140] Access Control Services
Capabilities: [150] Null
Capabilities: [200] Null
Kernel driver in use: pcieport
00:13.2 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 121
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
I/O behind bridge: [disabled] [16-bit]
Memory behind bridge: a1100000-a12fffff [size=2M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Capabilities: [a0] Power Management version 3
Capabilities: [100] Null
Capabilities: [140] Access Control Services
Capabilities: [150] Null
Capabilities: [200] Null
Kernel driver in use: pcieport
00:13.3 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3) (prog-if 00 [Normal decode])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 122
Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
I/O behind bridge: e000-efff [size=4K] [16-bit]
Memory behind bridge: a1300000-a13fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Capabilities: [a0] Power Management version 3
Capabilities: [100] Null
Capabilities: [140] Access Control Services
Capabilities: [150] Null
Capabilities: [200] Null
Kernel driver in use: pcieport
00:15.0 USB controller: Intel Corporation Celeron/Pentium Silver Processor USB 3.0 xHCI Controller (rev 03) (prog-if 30 [XHCI])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: medium devsel, IRQ 123
Memory at a1500000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Capabilities: [90] Vendor Specific Information: Len=14 <?>
Kernel driver in use: xhci_hcd
00:16.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 0 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 27
Memory at a152c000 (64-bit, non-prefetchable) [size=4K]
Memory at a152b000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:16.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 1 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 28
Memory at a152a000 (64-bit, non-prefetchable) [size=4K]
Memory at a1529000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:16.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 2 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 29
Memory at a1528000 (64-bit, non-prefetchable) [size=4K]
Memory at a1527000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:16.3 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 3 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at a1526000 (64-bit, non-prefetchable) [size=4K]
Memory at a1525000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:17.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 4 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 31
Memory at a1524000 (64-bit, non-prefetchable) [size=4K]
Memory at a1523000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:17.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 5 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 32
Memory at a1522000 (64-bit, non-prefetchable) [size=4K]
Memory at a1521000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:17.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 6 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 33
Memory at a1520000 (64-bit, non-prefetchable) [size=4K]
Memory at a151f000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:17.3 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor I2C 7 (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 34
Memory at a151e000 (64-bit, non-prefetchable) [size=4K]
Memory at a151d000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:19.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
DeviceName: Onboard LAN
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 35
Memory at a151c000 (64-bit, non-prefetchable) [size=4K]
Memory at a151b000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:19.1 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 36
Memory at a151a000 (64-bit, non-prefetchable) [size=4K]
Memory at a1519000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:19.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO SPI Host Controller (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 37
Memory at a1518000 (64-bit, non-prefetchable) [size=4K]
Memory at a1517000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] Vendor Specific Information: Len=14 <?>
00:1f.0 ISA bridge: Intel Corporation Celeron/Pentium Silver Processor LPC Controller (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, medium devsel, latency 0
00:1f.1 SMBus: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: medium devsel, IRQ 20
Memory at a1516000 (64-bit, non-prefetchable) [size=256]
I/O ports at f040 [size=32]
Kernel driver in use: i801_smbus
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd Device a801
Flags: bus master, fast devsel, latency 0, IRQ 22
Memory at a1400000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable+ Count=13 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [158] Power Budgeting <?>
Capabilities: [168] Secondary PCI Express
Capabilities: [188] Latency Tolerance Reporting
Capabilities: [190] L1 PM Substates
Kernel driver in use: nvme
05:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU (prog-if ff)
Subsystem: Global Unichip Corp. Coral Edge TPU
Flags: bus master, fast devsel, latency 0, IRQ 20
Memory at a1200000 (64-bit, prefetchable) [size=16K]
Memory at a1100000 (64-bit, prefetchable) [size=1M]
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [d0] MSI-X: Enable+ Count=128 Masked-
Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+
Capabilities: [f8] Power Management version 3
Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
Capabilities: [108] Latency Tolerance Reporting
Capabilities: [110] L1 PM Substates
Capabilities: [200] Advanced Error Reporting
Kernel driver in use: apex
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device b171
Flags: bus master, fast devsel, latency 0, IRQ 21
I/O ports at e000 [size=256]
Memory at a1304000 (64-bit, non-prefetchable) [size=4K]
Memory at a1300000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Capabilities: [170] Latency Tolerance Reporting
Capabilities: [178] L1 PM Substates
Kernel driver in use: r8169
Environment
- Talos version: [
talosctl version --nodes <problematic nodes>] 1.9.1 - Kubernetes version: [
kubectl version --short] 1.30.8 - Platform: bare metal
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (8 by maintainers)
Can you please do
talosctl get pcideviceson 1.8/1.9? Trying to see the diff