WSL: WSLg/Cuda suddenly broken due to nvidia-smi unable to find GPU
Version
10.0.22000.1098
WSL Version
- WSL 2
WSL 1
Kernel Version
5.15.68.1
Distro Version
Ubuntu 22.04
Other Software
WSL version: 0.70.5.0 WSLg version: 1.0.45 Direct3D version: 1.606.4 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Nvidia Driver: 526.47, Game Ready Driver, released 10/27/2022
Repro Steps
- Open wsl terminal
- Execute command
nvidia-smi
Expected Behavior
The nvidia-smi utility dumps diagnostic details about the GPU.
nvidia-smi.exe on Windows is able to display the expected output:
┖[~]> nvidia-smi
Tue Nov 1 10:18:07 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 526.47 Driver Version: 526.47 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... WDDM | 00000000:0B:00.0 Off | N/A |
| 0% 37C P8 17W / 350W | 1619MiB / 12288MiB | 8% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Actual Behavior
nvidia-smi on wsl/ubuntu 22.04 outputs a generic error instead:
dattebayo@<NGP'd>:~/dev$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Failed to properly shut down NVML: Driver Not Loaded
Diagnostic Logs
I’ll admit I’m kinda dumb when it comes to doing the linux diagnostics, which is part of what brought me here. Here’s what I’ve been able to gather from various Googlings and such though:
dpkg -l | grep nvidia
ii libnvidia-compute-495:amd64 510.85.02-0ubuntu0.22.04.1 amd64 Transitional package for libnvidia-compute-510
ii libnvidia-compute-510:amd64 510.85.02-0ubuntu0.22.04.1 amd64 NVIDIA libcompute package
rc libnvidia-compute-520:amd64 520.56.06-0ubuntu0.20.04.1 amd64 NVIDIA libcompute package
ii libnvidia-ml-dev:amd64 11.5.50~11.5.1-1ubuntu1 amd64 NVIDIA Management Library (NVML) development files
ii nvidia-cuda-dev:amd64 11.5.1-1ubuntu1 amd64 NVIDIA CUDA development files
ii nvidia-cuda-gdb 11.5.114~11.5.1-1ubuntu1 amd64 NVIDIA CUDA Debugger (GDB)
rc nvidia-cuda-toolkit 11.5.1-1ubuntu1 amd64 NVIDIA CUDA development toolkit
ii nvidia-cuda-toolkit-doc 11.5.1-1ubuntu1 all NVIDIA CUDA and OpenCL documentation
ii nvidia-opencl-dev:amd64 11.5.1-1ubuntu1 amd64 NVIDIA OpenCL development files
ii nvidia-profiler 11.5.114~11.5.1-1ubuntu1 amd64 NVIDIA Profiler for CUDA and OpenCL
ii nvidia-visual-profiler 11.5.114~11.5.1-1ubuntu1 amd64 NVIDIA Visual Profiler for CUDA and OpenCL
lsmod | grep nvidia
No output
(Truncated) DxDiag Output
------------------
System Information
------------------
Time of this report: 11/1/2022, 10:23:47
Machine name: NGP'd
Machine Id: {534C5435-EB65-464A-801F-79979E08B34E}
Operating System: Windows 11 Pro 64-bit (10.0, Build 22000) (22000.co_release.210604-1628)
Language: English (Regional Setting: English)
System Manufacturer: ASUS
System Model: System Product Name
BIOS: 3601 (type: UEFI)
Processor: AMD Ryzen 9 5950X 16-Core Processor (32 CPUs), ~3.4GHz
Memory: 131072MB RAM
Available OS Memory: 130980MB RAM
Page File: 53817MB used, 96617MB available
Windows Dir: C:\Windows
DirectX Version: DirectX 12
DX Setup Parameters: Not found
User DPI Setting: 144 DPI (150 percent)
System DPI Setting: 96 DPI (100 percent)
DWM DPI Scaling: Disabled
Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
DirectX Database Version: 1.2.2
DxDiag Version: 10.00.22000.0653 64bit Unicode
...
---------------
Display Devices
---------------
Card name: NVIDIA GeForce RTX 3080 Ti
Manufacturer: NVIDIA
Chip type: NVIDIA GeForce RTX 3080 Ti
DAC type: Integrated RAMDAC
Device Type: Full Device (POST)
Device Key: Enum\PCI\VEN_10DE&DEV_2208&SUBSYS_261219DA&REV_A1
Device Status: 0180200A [DN_DRIVER_LOADED|DN_STARTED|DN_DISABLEABLE|DN_NT_ENUMERATOR|DN_NT_DRIVER]
Device Problem Code: No Problem
Driver Problem Code: Unknown
Display Memory: Unknown
Dedicated Memory: n/a
Shared Memory: n/a
Current Mode: Unknown
HDR Support: Unknown
Display Topology: Unknown
Display Color Space: Unknown
Color Primaries: Unknown
Display Luminance: Unknown
Driver Name: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll,C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll,C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll,C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll
Driver File Version: 31.00.0015.2647 (English)
Driver Version: 31.0.15.2647
DDI Version: unknown
Feature Levels: Unknown
Driver Model: WDDM 3.0
Hardware Scheduling: DriverSupportState:Stable Enabled:True
Graphics Preemption: Pixel
Compute Preemption: Dispatch
Miracast: Not Supported by Graphics driver
Detachable GPU: No
Hybrid Graphics GPU: Discrete
Power P-states: Not Supported
Virtualization: Paravirtualization
Block List: No Blocks
Catalog Attributes: Universal:False Declarative:True
Driver Attributes: Final Retail
Driver Date/Size: 10/24/2022 5:00:00 PM, 772488 bytes
WHQL Logo'd: Yes
WHQL Date Stamp: Unknown
Device Identifier: Unknown
Vendor ID: 0x10DE
Device ID: 0x2208
SubSys ID: 0x261219DA
Revision ID: 0x00A1
Driver Strong Name: oem52.inf:0f066de3b91c4385:Section071:31.0.15.2647:pci\ven_10de&dev_2208
Rank Of Driver: 00CF2001
Video Accel: Unknown
DXVA2 Modes: Unknown
Deinterlace Caps: n/a
D3D9 Overlay: Unknown
DXVA-HD: Unknown
DDraw Status: Enabled
D3D Status: Not Available
AGP Status: Enabled
MPO MaxPlanes: Unknown
MPO Caps: Unknown
MPO Stretch: Unknown
MPO Media Hints: Unknown
MPO Formats: Unknown
PanelFitter Caps: Unknown
PanelFitter Stretch: Unknown
....
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 46
I meet same problem since Sep. and I can run cuda in docker in wsl2, but not in kali-linux I find that:
What works for me…
Add
/mnt/c/Windows/System32/lxss/libtoLD_LIBRARY_PATHas below:export LD_LIBRARY_PATH=/mnt/c/Windows/System32/lxss/lib:$LD_LIBRARY_PATHNB! The
lxss libpath must be before the other lib paths.This should fix the problem.
My environment:
and my .bashrc related to this:
Test on TensorFlow 2.13.0:
Do not mind NUMA warnings. WSL does not support NUMA at this time.
Enjoy your stable environment! At least, for the time being. 😃
谢谢兄弟 这玩意搞了我一个晚上 就离谱 头痛
During a program compilation, I need to link with a system library, so I set the environment variable like this:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnuI found that after this setting, nvidia-smi will not longer work and generate an error message similar to the one in this discussion. However, after removing this environment variable settings. Things can go back to normal gain.
A pretty irritating issue and I spent quite some time debugging it.
I have the same issue as described in #9134, so you are not alone.
I haven’t installed any external nvidia libraries in WSL either and I can run nvidia-smi.exe in WSL successfully but running the nvidia-smi located in
/usr/lib/wsl/lib/nvidia-smiand/mnt/c/Windows/System32/lxss/lib/nvidia-smiboth produce the same error as you have above.Manually shutting down and restarting doesn’t seem to yield any results either.
Make sure you set the distro to be the default in WSL.
this worked for me. Needless to say make sure nvidia-smi works fine from within Windows before trying any of the above
Thank you @gserbanut! After countless hours of trying to fix this (including installing/uninstalling cuda toolkit multiple times), your simple suggestions fixed my issue. Now both
nvidia-smiworks and PyTorch can see the gpuMy situation is the same as yours, it’s amazing!
@devttebayo it is not sound practice to add any
nvidia-utilspackages or drivers on WSL side. Thenvidia-smilocated at/usr/lib/wsl/lib/nvidia-smiis exported from the Windows side, being part of the Windows Nvidia driver. It is located atc:\windows\System32\lxss\lib\along with some Nvidia libraries used by WSL.Thanks, this works for me on WSL2 when having problem with
import torch;I have nvidia-smi installed only on Windows side, and had WSL installed with Administrator, that said, the user is literally “Administrator” and is an “Administrator” level account. I have also tried to create a “non-Admin” account, redo the entire WSL under that account, it still fails.
I’ve noticed a couple of similarities in these issues:
In my cases, the nvidia-smi only worked when exec from Windows Terminal as Admin.