WSL: nvidia-smi segmentation fault in wsl2 but not in Windows
Windows Version
10.0.22631.3235
WSL Version
2.1.4.0
Are you using WSL 1 or WSL 2?
- WSL 2
- WSL 1
Kernel Version
5.15.146.1-2
Distro Version
Ubuntu 22.04
Other Software
GeForce GTX 1650 Ti with GeForce Game Ready Driver version 551.76
Repro Steps
Run nvidia-smi in Windows and get the following:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 551.76 Driver Version: 551.76 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1650 Ti WDDM | 00000000:01:00.0 Off | N/A |
| N/A 63C P8 3W / 50W | 163MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 3268 C+G ...ekyb3d8bbwe\WsaClient\WsaClient.exe N/A |
| 0 N/A N/A 18112 C+G ...ience\NVIDIA GeForce Experience.exe N/A |
+-----------------------------------------------------------------------------------------+
Run nvidia-smi in wsl2 Ubuntu and get the following:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06 Driver Version: 551.76 CUDA Version: 12.4 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
[1] 2058 segmentation fault nvidia-smi
Expected Behavior
I am expecting no segmentation fault and successful output in WSL 2.
Actual Behavior
I get a segmentation fault in WSL2 as described above.
Diagnostic Logs
No response
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 26
- Comments: 85
I still see the segfault.
However, if I run nvidia-smi.exe from within WSL it displays correctly.
Additionally, if I try running programs that use CUDA they do run.
Tried a few different versions and it seems like everything 538+ is broken.
Windows version of NVIDIA GPU driver 565.90, CUDA level up to 12.7
The nvidia-smi segmentation fault issue on some GPUs in the WSL2 environment, which has persisted since version 538.15, has been fixed in 565.90.
This version also fixes the startup crash of Dying Light 2: Enhanced Edition.
Shut down the wsl with
wsl --shutdownand rerun it work for mesame, downgrade to 537 solve my problem.
This is a problem that has not been solved since 2024 (538 and subsequent versions) until now
The 537.96 version (Geforce Vulkan BETA, supports CUDA12.2 at most) works normally. The download link is attached
https://developer.nvidia.com/downloads/vulkan-beta-53796-windows )
To add:I found that only the desktop versions of GTX 1050Ti , RTX 2000 , RTX 3050 and RTX 4070 do not have this issue with me. I have asked Linux kernel development engineers, and they said that this issue needs to be fixed.
good news: find solution. bad news: RTX 4080 super has no previous driver to downgrade. sad
I uninstalled the NVIDIA driver and installed v537.58 as advised in the last few days, and the Segmentation fault on WSL2 disappeared. Thanks for the replies guys!
same, downgrade to 537 solve my problem.
I think it is the issue with NVIDIA 551 driver. It works for me with the previous NVIDIA 537 but after upgrading, I got segmentation fault in WSL2 as well.
same issue but even less information in WSL2, nothing printed out rather than Segmentation fault, not even heads including version
Still problem exists with 560 drivers
@kziemski I thought so too as the first thing I did after installing WSL2 was to
nvidia-smito check for the presence of the GPU. Turns out, onlynvidia-smigives errors, everything else seems to work normally.Had the same issue and it kept me up for very long… the thing that fixed it for me was uninstalling the Nvidia driver (which I updated to 551.76 ) and installing an older one (NOT 551.61 which also didn’t work), 537.58 from October 2023 in my case but it was pretty random choice. (I have the MX550 so the drivers correspond to that GPU model). hope it will help you as well.
Hi I’m an AI powered bot that finds similar issues based off the issue title.
Please view the issues below to see if they solve your problem, and if the issue describes your problem please consider closing this one and thumbs upping the other issue to help us prioritize it. Thank you!
Open similar issues:
Closed similar issues:
Can someone from Microsoft work with Nvidia to fix this issue? My work laptop auto updated the driver to 555.99 and this issue came back again. GPU is RTX A2000. Happens in Podman container as well as Ubuntu on WSL2
confirmed I still get the
nvidia-smiSegmentation Fault in WSL2 on the latest driver:otherwise, the GPU workloads I tested on WSL2 are working (also via docker) and
nvidia-smiworks from PowerShell@kziemski interesting, so far, all versions (538, 54x, 555, 559) work for me but I’m on a RTX/Quadro not Geforce series.
In my experience, it’s a hit or miss issue depending on the nvidia driver version. With
wsl-2.3.11andnvidia-555.99it works:However, there were times when nvidia-smi would segfault. It only takes one update for it to fail again!
Dell XPS15 9500 - WSL2 - Nvidia GTX1650TI - GRD537.13 - Ubuntu 24.04 works
It works perfectly, thank you!
So far here is my solution: as of Apr-13-2024 I have downgraded from 551.86-quadro-rtx-desktop-notebook-win10-win11-64bit-international-dch-whql (BUILT on CUDA 12.4) down to 536.96-quadro-rtx-desktop-notebook-win10-win11-64bit-international-dch-whql (BUILT on CUDA 12.2) and segmentation fault error dissapeared. Seems to me problem is that Windows Ubuntu WSL2 image does not support latest NVIDIA gears.
Don’t work for me @Rui-K
At least in my case, I can confirm this solved the problem for me.
Download the latest drivers from Nvidia, the “Latest” one is only the latest stable version - there are betas / New Feature Branch builds out there that are newer but maybe not as stable / tested:
This solved the problem for me
In my case the problem seems to be associated with libnvidia-ml.so.1 file:
You can check the GPU details with below command as well. This is working fine for me on WSL2.
o/p of command = GPU 0: NVIDIA GeForce RTX 2050 (UUID: GPU-2b9a833b-6fb9-b25e-1833-7ff832a835eb)
Also if you want to install cuda toolkit simply install pytorch. This package would install cuda toolkit automatically for GPU.
Apparently, this is a feature not a bug. NVIDIA-SMI support is limited… although there is not mention of Segmentation fault you might be seeing in the following docs: https://docs.nvidia.com/cuda/wsl-user-guide/index.html
@jsvetrifork I can confirm that 538 does not solve the problem. What is interesting is that I can, even if there is a segmentation fault in SMI, still use it in Python:
This is a problem that has not been solved since 2024 (538 and subsequent versions) until now
This method is invalid
I was having exactly the same problem. I solved it by creating a ‘.wslconfig’ file in 'C:\Users<yourUserName>' with: [wsl2] memory=4GB processors=2 gpu=true
I pulled down to Game Ready 537.13 and it worked. Thank you.
As said before just install the “normal” windows 537 driver or earlier. The Download Page has older driver versions available. If you try to install cuda for windows, it will bump the driver to the latest version. In my case, I didn’t have to purge the windows driver first, just running the installer for 537.xx (it doesn’t matter which one Nvidia displays for you).
The same error occurred. Nvidia RTX A500 Notebook GPU. When I downgraded from R550 U5 (552.22) to R535 U11 (538.33), the segmentation fault disappeared.
@eyabesbes you have to uninstall the current Nvidia Windows driver then install version 537. Nvidia WSL libraries are part of the Windows installer.
Nvidia studio drivers seems to work okay:
Hi, I’m having the same issue on WSL2 but not on Windows11 how can I downgrade to 537 ?
Tue Apr 9 20:40:24 2024 ±----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.54.10 Driver Version: 551.61 CUDA Version: 12.4 | |-----------------------------------------±-----------------------±---------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| Segmentation fault
Update: Did nothing, reboot my computer, without trying nvidia-smi in windows, I directly tried it in WSL, worked with no error.
Same problem here…
nvidia-smiworks fine on Win11, but gives Segmentation fault on WSL2@themizzi which
nvidia-smiare you running inside WSL? The correct one is part of the Nvidia Windows driver installation and is mounted in WSL under/usr/lib/wsl/lib. What is the output ofcommand -v nvidia-smiin WSL?Output using Game Ready Driver ver. 551.67:
Notice the nvidia-smi version, it’s
550.60.01