WSL: WSLg/Cuda suddenly broken due to nvidia-smi unable to find GPU

Version

10.0.22000.1098

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

5.15.68.1

Distro Version

Ubuntu 22.04

Other Software

WSL version: 0.70.5.0 WSLg version: 1.0.45 Direct3D version: 1.606.4 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp

Nvidia Driver: 526.47, Game Ready Driver, released 10/27/2022

Repro Steps

  1. Open wsl terminal
  2. Execute command nvidia-smi

Expected Behavior

The nvidia-smi utility dumps diagnostic details about the GPU.

nvidia-smi.exe on Windows is able to display the expected output:

┖[~]> nvidia-smi
Tue Nov  1 10:18:07 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 526.47       Driver Version: 526.47       CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:0B:00.0 Off |                  N/A |
|  0%   37C    P8    17W / 350W |   1619MiB / 12288MiB |      8%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Actual Behavior

nvidia-smi on wsl/ubuntu 22.04 outputs a generic error instead:

dattebayo@<NGP'd>:~/dev$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Failed to properly shut down NVML: Driver Not Loaded

Diagnostic Logs

I’ll admit I’m kinda dumb when it comes to doing the linux diagnostics, which is part of what brought me here. Here’s what I’ve been able to gather from various Googlings and such though:

dpkg -l | grep nvidia

ii  libnvidia-compute-495:amd64     510.85.02-0ubuntu0.22.04.1              amd64        Transitional package for libnvidia-compute-510
ii  libnvidia-compute-510:amd64     510.85.02-0ubuntu0.22.04.1              amd64        NVIDIA libcompute package
rc  libnvidia-compute-520:amd64     520.56.06-0ubuntu0.20.04.1              amd64        NVIDIA libcompute package
ii  libnvidia-ml-dev:amd64          11.5.50~11.5.1-1ubuntu1                 amd64        NVIDIA Management Library (NVML) development files
ii  nvidia-cuda-dev:amd64           11.5.1-1ubuntu1                         amd64        NVIDIA CUDA development files
ii  nvidia-cuda-gdb                 11.5.114~11.5.1-1ubuntu1                amd64        NVIDIA CUDA Debugger (GDB)
rc  nvidia-cuda-toolkit             11.5.1-1ubuntu1                         amd64        NVIDIA CUDA development toolkit
ii  nvidia-cuda-toolkit-doc         11.5.1-1ubuntu1                         all          NVIDIA CUDA and OpenCL documentation
ii  nvidia-opencl-dev:amd64         11.5.1-1ubuntu1                         amd64        NVIDIA OpenCL development files
ii  nvidia-profiler                 11.5.114~11.5.1-1ubuntu1                amd64        NVIDIA Profiler for CUDA and OpenCL
ii  nvidia-visual-profiler          11.5.114~11.5.1-1ubuntu1                amd64        NVIDIA Visual Profiler for CUDA and OpenCL

lsmod | grep nvidia

No output

(Truncated) DxDiag Output

------------------
System Information
------------------
      Time of this report: 11/1/2022, 10:23:47
             Machine name: NGP'd
               Machine Id: {534C5435-EB65-464A-801F-79979E08B34E}
         Operating System: Windows 11 Pro 64-bit (10.0, Build 22000) (22000.co_release.210604-1628)
                 Language: English (Regional Setting: English)
      System Manufacturer: ASUS
             System Model: System Product Name
                     BIOS: 3601 (type: UEFI)
                Processor: AMD Ryzen 9 5950X 16-Core Processor             (32 CPUs), ~3.4GHz
                   Memory: 131072MB RAM
      Available OS Memory: 130980MB RAM
                Page File: 53817MB used, 96617MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 144 DPI (150 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.2.2
           DxDiag Version: 10.00.22000.0653 64bit Unicode
...
---------------
Display Devices
---------------
           Card name: NVIDIA GeForce RTX 3080 Ti
        Manufacturer: NVIDIA
           Chip type: NVIDIA GeForce RTX 3080 Ti
            DAC type: Integrated RAMDAC
         Device Type: Full Device (POST)
          Device Key: Enum\PCI\VEN_10DE&DEV_2208&SUBSYS_261219DA&REV_A1
       Device Status: 0180200A [DN_DRIVER_LOADED|DN_STARTED|DN_DISABLEABLE|DN_NT_ENUMERATOR|DN_NT_DRIVER] 
 Device Problem Code: No Problem
 Driver Problem Code: Unknown
      Display Memory: Unknown
    Dedicated Memory: n/a
       Shared Memory: n/a
        Current Mode: Unknown
         HDR Support: Unknown
    Display Topology: Unknown
 Display Color Space: Unknown
     Color Primaries: Unknown
   Display Luminance: Unknown
         Driver Name: C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll,C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll,C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll,C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ade64cd54ec2f9ed\nvldumdx.dll
 Driver File Version: 31.00.0015.2647 (English)
      Driver Version: 31.0.15.2647
         DDI Version: unknown
      Feature Levels: Unknown
        Driver Model: WDDM 3.0
 Hardware Scheduling: DriverSupportState:Stable Enabled:True 
 Graphics Preemption: Pixel
  Compute Preemption: Dispatch
            Miracast: Not Supported by Graphics driver
      Detachable GPU: No
 Hybrid Graphics GPU: Discrete
      Power P-states: Not Supported
      Virtualization: Paravirtualization 
          Block List: No Blocks
  Catalog Attributes: Universal:False Declarative:True 
   Driver Attributes: Final Retail
    Driver Date/Size: 10/24/2022 5:00:00 PM, 772488 bytes
         WHQL Logo'd: Yes
     WHQL Date Stamp: Unknown
   Device Identifier: Unknown
           Vendor ID: 0x10DE
           Device ID: 0x2208
           SubSys ID: 0x261219DA
         Revision ID: 0x00A1
  Driver Strong Name: oem52.inf:0f066de3b91c4385:Section071:31.0.15.2647:pci\ven_10de&dev_2208
      Rank Of Driver: 00CF2001
         Video Accel: Unknown
         DXVA2 Modes: Unknown
      Deinterlace Caps: n/a
        D3D9 Overlay: Unknown
             DXVA-HD: Unknown
        DDraw Status: Enabled
          D3D Status: Not Available
          AGP Status: Enabled
       MPO MaxPlanes: Unknown
            MPO Caps: Unknown
         MPO Stretch: Unknown
     MPO Media Hints: Unknown
         MPO Formats: Unknown
    PanelFitter Caps: Unknown
 PanelFitter Stretch: Unknown
....

About this issue

Most upvoted comments

I meet same problem since Sep. and I can run cuda in docker in wsl2, but not in kali-linux I find that:

  • If I start from Windows Terminal(Admin): nvidia-smi will fail like this
  • If I start from Windows Terminal(without Admin): nvidia-smi sucess

What works for me…

OS: Windows 11 GPU: NVIDIA GeForce GTX 1650 WSL: 2 WSL distro: Ubuntu 20.04 (it should work on 22.04 as well) NVIDIA drivers: R536 CUDA: 11.8 (required by TensorFlow 2.13.0) cuDNN: 8.9 (compatibility requirement with CUDA 11.8) TensorRT: 8.6 (compatibility requirement with CUDA 11.8)

Add /mnt/c/Windows/System32/lxss/lib to LD_LIBRARY_PATH as below: export LD_LIBRARY_PATH=/mnt/c/Windows/System32/lxss/lib:$LD_LIBRARY_PATH

NB! The lxss lib path must be before the other lib paths.

This should fix the problem.

My environment:

(tf2) root@DESKTOP-WIN11:~# cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/include"
LD_LIBRARY_PATH="/mnt/c/Windows/System32/lxss/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:/usr/lib/x86_64-linux-gnu"

and my .bashrc related to this:

(tf2) root@DESKTOP-WIN11:~# grep environment .bashrc
set -a; source /etc/environment; set +a;

Test on TensorFlow 2.13.0:

(tf2) root@DESKTOP-WIN11:~# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2023-08-18 09:06:27.738396: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-18 09:06:31.057671: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-08-18 09:06:31.096912: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-08-18 09:06:31.098038: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Do not mind NUMA warnings. WSL does not support NUMA at this time.

Enjoy your stable environment! At least, for the time being. 😃

I meet same problem since Sep. and I can run cuda in docker in wsl2, but not in kali-linux I find that:

  • If I start from Windows Terminal(Admin): nvidia-smi will fail like this
  • If I start from Windows Terminal(without Admin): nvidia-smi sucess

谢谢兄弟 这玩意搞了我一个晚上 就离谱 头痛

During a program compilation, I need to link with a system library, so I set the environment variable like this: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu

I found that after this setting, nvidia-smi will not longer work and generate an error message similar to the one in this discussion. However, after removing this environment variable settings. Things can go back to normal gain.

A pretty irritating issue and I spent quite some time debugging it.

I have the same issue as described in #9134, so you are not alone.

I haven’t installed any external nvidia libraries in WSL either and I can run nvidia-smi.exe in WSL successfully but running the nvidia-smi located in /usr/lib/wsl/lib/nvidia-smi and /mnt/c/Windows/System32/lxss/lib/nvidia-smi both produce the same error as you have above.

Manually shutting down and restarting doesn’t seem to yield any results either.

Make sure you set the distro to be the default in WSL.

  1. wsl --shutdown
  2. wsl --setdefault <Distro Name> then start the distro wsl -d <Distro Name> and run nvidia-smi

this worked for me. Needless to say make sure nvidia-smi works fine from within Windows before trying any of the above

Add /mnt/c/Windows/System32/lxss/lib to LD_LIBRARY_PATH as below: export LD_LIBRARY_PATH=/mnt/c/Windows/System32/lxss/lib:$LD_LIBRARY_PATH NB! The lxss lib path must be before the other lib paths.

Thank you @gserbanut! After countless hours of trying to fix this (including installing/uninstalling cuda toolkit multiple times), your simple suggestions fixed my issue. Now both nvidia-smi works and PyTorch can see the gpu

I meet same problem since Sep. and I can run cuda in docker in wsl2, but not in kali-linux I find that:

  • If I start from Windows Terminal(Admin): nvidia-smi will fail like this
  • If I start from Windows Terminal(without Admin): nvidia-smi sucess

My situation is the same as yours, it’s amazing!

@devttebayo it is not sound practice to add any nvidia-utils packages or drivers on WSL side. The nvidia-smi located at /usr/lib/wsl/lib/nvidia-smi is exported from the Windows side, being part of the Windows Nvidia driver. It is located at c:\windows\System32\lxss\lib\ along with some Nvidia libraries used by WSL.

What works for me…

OS: Windows 11 GPU: NVIDIA GeForce GTX 1650 WSL: 2 WSL distro: Ubuntu 20.04 (it should work on 22.04 as well) NVIDIA drivers: R536 CUDA: 11.8 (required by TensorFlow 2.13.0) cuDNN: 8.9 (compatibility requirement with CUDA 11.8) TensorRT: 8.6 (compatibility requirement with CUDA 11.8)

Add /mnt/c/Windows/System32/lxss/lib to LD_LIBRARY_PATH as below: export LD_LIBRARY_PATH=/mnt/c/Windows/System32/lxss/lib:$LD_LIBRARY_PATH

NB! The lxss lib path must be before the other lib paths.

This should fix the problem.

My environment:

(tf2) root@DESKTOP-WIN11:~# cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/include"
LD_LIBRARY_PATH="/mnt/c/Windows/System32/lxss/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:/usr/lib/x86_64-linux-gnu"

and my .bashrc related to this:

(tf2) root@DESKTOP-WIN11:~# grep environment .bashrc
set -a; source /etc/environment; set +a;

Test on TensorFlow 2.13.0:

(tf2) root@DESKTOP-WIN11:~# python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2023-08-18 09:06:27.738396: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-18 09:06:31.057671: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-08-18 09:06:31.096912: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-08-18 09:06:31.098038: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Do not mind NUMA warnings. WSL does not support NUMA at this time.

Enjoy your stable environment! At least, for the time being. 😃

Thanks, this works for me on WSL2 when having problem with import torch;

Just trying to link all the relevant threads here: canonical/microk8s#3024 #9254 #8174 #9134 This cannot be a coincident.

@fzhan

nvidia-smi needs to be from the Windows driver package. It is mapped to /usr/lib/wsl/lib/nvidia-smi,

There is an issue when nvidia-smi and other Cuda applications are running from a WSL window, started as Administrator or not. For example, if you start WSL as Administrator the very first time after boot, nvidia-smi works. If you start another WSL as non-Admininstator, it fails. The opposite is also true, If you start WSL the very first time as non-Admininstator, nvidia -smi works. If you start another WSL window as Admininstator, nvidia -smi fails. This is under investigation. It might be related to your case.

I have nvidia-smi installed only on Windows side, and had WSL installed with Administrator, that said, the user is literally “Administrator” and is an “Administrator” level account. I have also tried to create a “non-Admin” account, redo the entire WSL under that account, it still fails.

I’ve noticed a couple of similarities in these issues:

  1. latest Windows 11 with WSL up-to-date.
  2. latest nvidia driver, or at least after 520.
  3. cuda 11.8 or 12 (being the latest)
  4. wsl-ubuntu with the instruction listed on nvidia website, but some are using it on Ubuntu-22.04 where the instruction has 20.04

In my cases, the nvidia-smi only worked when exec from Windows Terminal as Admin.