WSL: WSL2 & CUDA does not work [v20226]
Environment
Windows build number: 10.0.20226.0
Your Distribution version: 18.04 / 20.04
Whether the issue is on WSL 2 and/or WSL 1: Linux version 4.19.128-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Tue Jun 23 12:58:10 UTC 2020
Steps to reproduce
Exactly followed instructions available at https://docs.nvidia.com/cuda/wsl-user-guide/index.html Tested on previously working Ubuntu WSL image (IIRC GPU last worked on 20206, than whole WSL2 stopped working) Tested also on newly created Ubuntu 18.04 and Ubuntu 20.04 images.
I have tested CUDA compatible NVIDIA drivers 455.41 & 460.20. I have tried removing all drivers etc. I have also tested using CUDA 10.2 & CUDA 11.0.
It was tested on two separate machines (one Intel + GTX1060, other Ryzen + RTX 2080Ti)
Issue tested directly in OS also in docker containers inside.
Example (directly in Ubuntu):
piotr@DESKTOP-FS6J3NT:/usr/local/cuda/samples/4_Finance/BlackScholes$ ./BlackScholes
[./BlackScholes] - Starting...
GPU Device 0: "Turing" with compute capability 7.5
Initializing data...
...allocating CPU memory for options.
...allocating GPU memory for options.
CUDA error at BlackScholes.cu:116 code=46(cudaErrorDevicesUnavailable) "cudaMalloc((void **)&d_CallResult, OPT_SZ)"
Example in container:
piotr@DESKTOP-FS6J3NT:/mnt/c/Users/pppnn$ docker run -it --gpus all -p 8888:8888 tensorflow/tensorflow:latest-gpu-py3-jupyter python
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2020-10-01 14:18:07.538627: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-10-01 14:18:07.624188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
>>> tf.test.is_gpu_available()
WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2020-10-01 14:18:32.359457: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-01 14:18:32.398949: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3200035000 Hz
2020-10-01 14:18:32.402692: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d06b70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-01 14:18:32.402748: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-10-01 14:18:32.409370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-10-01 14:18:32.877228: W tensorflow/compiler/xla/service/platform_util.cc:276] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2020-10-01 14:18:32.877370: I tensorflow/compiler/jit/xla_gpu_device.cc:136] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA
2020-10-01 14:18:32.879904: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:32.880192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1d:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.665GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-01 14:18:32.880277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-01 14:18:32.880340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-01 14:18:32.959947: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-01 14:18:32.973554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-01 14:18:33.111736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-01 14:18:33.127902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-01 14:18:33.128018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-01 14:18:33.128535: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:33.129170: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:33.129403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-10-01 14:18:33.131671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/test_util.py", line 1513, in is_gpu_available
for local_device in device_lib.list_local_devices():
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/device_lib.py", line 43, in list_local_devices
_convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable
>>>
>>>
>>>
>>>
>>> tf.config.list_physical_devices('GPU')
2020-10-01 14:18:55.610151: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:55.610510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1d:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.665GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-01 14:18:55.610579: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-01 14:18:55.610623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-01 14:18:55.610676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-01 14:18:55.610719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-01 14:18:55.610762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-01 14:18:55.610805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-01 14:18:55.610846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-01 14:18:55.611251: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:55.611765: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:55.611999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>>>
>>>
>>>
>>> tf.test.gpu_device_name()
2020-10-01 14:20:08.762060: W tensorflow/compiler/xla/service/platform_util.cc:276] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2020-10-01 14:20:08.762222: I tensorflow/compiler/jit/xla_gpu_device.cc:136] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA
2020-10-01 14:20:08.762863: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:20:08.763201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1d:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.665GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-01 14:20:08.763263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-01 14:20:08.763316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-01 14:20:08.763358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-01 14:20:08.763379: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-01 14:20:08.763428: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-01 14:20:08.763480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-01 14:20:08.763533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-01 14:20:08.763898: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:20:08.764536: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:20:08.764810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/test_util.py", line 112, in gpu_device_name
for x in device_lib.list_local_devices():
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/device_lib.py", line 43, in list_local_devices
_convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable
>>>
Expected behavior
CUDA working inside WSL2
Actual behavior
All tests which are using CUDA inside WSL Ubuntu are resulting with various CUDA errors - mostly referring to no CUDA devices available.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 75
- Comments: 176 (5 by maintainers)
We have identified the issue and are working on a fix.
Build 20236 blog post annonuncement
Can confirm that this is currently still happening in the most recent version – 20231
I confirm the same issue exists in 20231.1000
I am having the same problem. WIndows 10 build 20226 and Nvidia driver 460.20. It is great to see that it is not just my install. I hope that this can be fixed soon.
And now I can also confirm that it will work if you roll back to the previous build 20221. You can download the (old) iso file from Microsoft and re-install without losing any data.
After doing this, it still didn’t work for me.
I was able to get it to work by following it up with this:
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pubsudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'sudo apt-get updatesudo apt-get install -y cuda-toolkit-11-1wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.shbash Miniconda3-latest-Linux-x86_64.shconda config --set auto_activate_base falseconda create --name directml python=3.6conda activate directmlpip install tensorflow-directmlimport tensorflow.compat.v1 as tftf.enable_eager_execution(tf.ConfigProto(log_device_placement=True))print(tf.add([1.0, 2.0], [3.0, 4.0]))And there, I finally saw the name of my GPU pop up!
After that I went back and tried the ./BlackScholes test:
cd /usr/local/cuda/samples/4_Finance/BlackScholessudo make./BlackScholesAnd it also worked!
Seems this issue has been tagged “fix inbound” for quite a while, but is not even mentioned in the 20231 release note(not even in the known issues part).
Could more attention be paid to this issue? This is the main blocker for me as a CUDA programmer.
It is also fixed for me in 20236 so I am closing the issue. Thanks to all commenters and also WSL developers for fixing the issue.
Cuda on WSL2 works perfectly on build 20236. The problem seems to be resolved.
I am having the same issue. Everything was working flawlessly this morning but then I have updated to 20226.1000 from 20221.1000 and it does not work anymore (tried reinstalling nvidia drivers, etc.) with error that all cuda devices are busy or unavailable.
Edit: After going back to version 20221, everything works again, thus it confirms that the new version caused the problem.
@ccs96307 Now what can we say phew… solving 1 problem leads to another
With Windows 10 (non-insider) 2021H2 November update, CUDA now works in WSL 2.
Same issue here on the 20231 version , this issue needs to be fixed soon!
Error on WSL2
Environment: Windows insider program 21332.1000, NVIDIA Driver 461.72 on Windows (for GeForce GTX 1660 Ti) , WSL2 Ubuntu 20-04, Error: CUDA error at …/…/common/inc/helper_cuda.h:779 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”
After installing and having the NVIDIA driver working on Windows. I followed the NVIDA site https://docs.nvidia.com/cuda/wsl-user-guide/index.html instructions (adapted for Ubuntu 20-04 because the example uses 18-04)
Steps on Ubuntu-20.04:
After compiling and executing the ./BlackScholes example I get (again) the same error described on this thread. Although there is a comment earlier that it worked again for user EtienneT with the Windows insider release 21332.1000, it didn’t work for me. It does the same as with the previous insider release 21327.1000. Note that Microsoft release notes for 21327 admits than GPU computing won’t work (a regression bug), but in the release notes for 21332 MS says it is fixed. I get the same error on both Windows insider releases.
Situation on Windows 10
However the GPU works on Windows with using Pytorch
Pytorch Windows, Installed with: conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
Code: print(torch. cuda. get_device_name(torch. cuda. current_device())) It prints: GeForce GTX 1660 Ti with Max-Q Design
But tensorflow 2.4.1 on Windows installed with: pip install tensorflow It recognizes the GPU but fails downstream with a known problem with Tensorflow for which I cannot find updated info. Code: if tf.test.gpu_device_name() != ‘/device:GPU:0’: print(‘WARNING: GPU device not found.’) else: print(‘SUCCESS: Found GPU: {}’.format(tf.test.gpu_device_name())) physical_devices = tf.config.list_physical_devices(‘GPU’) tf.config.experimental.set_memory_growth(physical_devices[0], True) #thia should solve the cuda solver problem
It prints ‘SUCCESS: Found GPU:’ etc, but later in the process it has a problem in one of the cuda libraries. Here is the output of the run:
Note that for Tensorflow I had to download the CUDA Toolk kit 11.2 and associated Cudadnn libraries.
Any help will be appreciated. Petrarca Bruto
Windows insider updated to 20262.1 With the update, the regular nvidia driver was re-installed. Not good for wsl2. Also, nvidia issued a new driver that you should install in windows: https://developer.nvidia.com/cuda/wsl The new drive is 465.12 Download and install it. You must reboot after the install or wsl2 will not see the gpu as of yet. Start wsl2
Then do the above docker procedures to check that the docker aslo works. In my case, it works fine.
Build 20236 fixed it for me 👍
Same problem on 20231. Cuda samples error out, for example matrixMul:
I hope this is fixed soon enough
@mitchellvitez I really don’t think going back to 20201 is safe move considering previous bugs in 20201 as well
I have the error
remote procedure call failedin the last version, and I have this issue after upgrade. So…when I recovery does it mean I will get theremote procedure call failedback 😿Yes, you can install 20221 from https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewadvanced
It’s three days I continue trying different solutions but seems there’s none apart from downgrading… I get the same error as the above posts either with cuda-toolkit-11-0 and cuda toolkit-11-1 with my small nvidia GeForce MX150. The error constantly thrown up is (for the .BlackScholes example given in the CUDA on WSL guide https://docs.nvidia.com/cuda/wsl-user-guide/index.html) is:
CUDA error at BlackScholes.cu:116 code=46(cudaErrorDevicesUnavailable) "cudaMalloc((void **)&d_CallResult, OPT_SZ)"I bookmarked this page and following updates daily!I have the same problem in 20226.
Cuda device query is ok.
But cuda utility does not worked.
Below is strace log. BlackScholes_cuda_error_log.zip
Oh too bad, I also encountered this problem. I was so happy when wsl worked again in the 20226 version, but cuda couldn’t work. I was left out of the cold. I tried the following solutions, but none of them worked for me.
Reinstall the graphics card driver 460.20.
Recompile cuda dependent environment library.
Uninstall wsl2 and kernel program and reinstall.
I just updated to
20262.1010and reinstalled nvidia driver465.12, and installed Ubuntu 20.04 WSL2, and setup accordingly (used the CUDA WSL2 tutorial and replaced 1804 with 2004), and it just worked ! and even my previous Ubuntu 18.04 started working for CUDA (it wasn’t working on older build), Thanks to @serg06 !I downgraded the version to 20221, the WSL 2 I installed showed the error message “remote procedure call failed”.
Same issue here. If you use the Build 20231, I don’t suggest to downgrade because it introduces a new error about user accounts. If you want to try, please consider backup.
@kivancguckiran I just joined the insider build so I’m in the same boat. It would probably take like 4 hours, but you could probably revert windows to the previous version (non-insider) maybe then go specifically to 20221. I’m not going to try it and just wait though.
@kerim371 I’m afraid it won’t work.
https://docs.nvidia.com/cuda/wsl-user-guide/index.html
What could I do if I Nvidia driver manager can’t find appropriate driver for me? Samsung RC720 notebook OS build: 21370.1 Basic GPU is Intel, additional Nvidia GeForce GT 520M According to Nvidia docs my GeForce GT 520M supports CUDA
Windows insider updated to 21327.1000 cuda broken again. previous insider version work well.
Error message with BlackScholes:
CUDA error at ../../common/inc/helper_cuda.h:779 code=35(cudaErrorInsufficientDriver) "cudaGetDeviceCount(&device_count)After re-install(Custom-Perform a clean install-restart pc) the CUDA driver (wsl ready) all goes well again.
This makes it work. Basically install docker on wsl2. https://dev.to/bartr/install-docker-on-windows-subsystem-for-linux-v2-ubuntu-5dl7
After restart of wsl2 do the following procedure:
One more thing you need to know. The usual docker-compose (last one is version 1.27) that gets installed with the docker does not support starting dockers with gpu. The only way to manage docker containers with gpu is using an experimental version of docker-compose that does support management of docker containers that use a gpu.
I confirm the issue is resolved in 20236.1000. Thanks to all who contributed momentum toward this.
Same issue for me on Build 20231, WSL2 and GTX 1650:
No GPU is listed after issuing the command lspci on Ubuntu 18.04:
The Tensorflow function list_local_devices returns:
Build 20201 should be a good build stop too. CUDA in WSL2 works since build 20145.
Previous builds in the near past trigger another fatal bug, while ancient ones don’t even deliver the needed functionality. That’s why I’m checking this thread constantly. Once I can run my first CUDA program, I’ll freeze my dev channel gladly, but sadly not now.
I hope so, @onomatopellan !
I froze my automatic update for 7 days. NVIDIA just released driver 456.71 I keep my 460.20.
By the way, Nvidia’s “default” install is CUDA 11.1 and cuDNN 8.0 most software requires earlier versions that do work perfectly well.
For example, there is no mxnet for cuda 11.x.
Here is how you install a previous version: Start here: (do not fall into the trap of installation for Ubuntu. Crack the cryptic installation for WSL !) https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#wsl-x86_64-rpm
Find the tensorflow-gpu version you want and the cuda version that it needs and the cudnn you need. https://www.tensorflow.org/install/gpu (I use tensorflow-gpu 1.14)
Run the below to get your disribution $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) $ echo $distribution (n my case it gives: ubuntu18.04)
Go into: https://developer.download.nvidia.com/compute/cuda/repos/ and click on your distribution (in my case: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/)
Get the cuda version you want (in my case https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-10-0_10.0.130-1_amd64.deb)
Becomes: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
Get the pin:
wget https://developer.download.nvidia.com/compute/cuda/repos/<distro>/<architecture>/cuda-<distro>.pin $ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin cuda-ubuntu1804.pin 100%[===================>] 190 --.-KB/s in 0s
Saving to: ‘cuda-ubuntu1804.pin’ 2020-10-05 23:20:15 (14.3 MB/s) - ‘cuda-ubuntu1804.pin’ saved [190/190]
sudo dpkg -i cuda-repo-<distro><version><architecture>.deb
sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo mv cuda-<distro>.pin /etc/apt/preferences.d/cuda-repository-pin-600 $ sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 $ sudo apt-get update $ sudo apt-get install cuda=10.0.130-1 dpkg: warning: downgrading cuda from 11.1.0-1 to 10.0.130-1 (Reading database … 190270 files and directories currently installed.) Preparing to unpack …/cuda_10.0.130-1_amd64.deb … Unpacking cuda (10.0.130-1) over (11.1.0-1) … Setting up cuda (10.0.130-1) …
add to ~/.bashrc (need to be done once in a lifetime. All cuda installs you would do later create symbolic links to these paths) export PATH=/usr/local/cuda/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
install cudnn 7.6 (select yours in the nvidia repo)
you need to install both regular and -dev versions
http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/
we need cuda-10.0 and cudnn 7.6.5 (you can choose another cudnn if needed from the cudnn-repo) http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb $ dpkg -i libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb $ dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb
check your installation $ python
Note: if for some reason your apt-geu update/upgade/dist-upgrade cause cuda 11.x to be installed over your previous version, simply run the three dpkg -i on cuda, cudnn and cudnn-dev and your version will be back on line. Usually nothing eles is needed.
The following worked for me:
This is called out in the flight notes: https://blogs.windows.com/windows-insider/2021/03/03/announcing-windows-10-insider-preview-build-21327/
Updated to Insider version 20270.1 still with 465.12 and all is working well as before.
I can confirm that installing the latest docker edge, Microsoft Edge packages, and NVIDIA 465.12 makes it all work again.
I had stopped working on this for a month because an update killed it,it was badly broken.
Fortunately it is working again.
Even better than before actually because now NVIDIA DOCKER can run in multiple containers simultaneously!!!
Thanks for posting your steps. I also run Ubuntu in WSL2 and use the Nvidia docker containers as explained in https://docs.nvidia.com/cuda/wsl-user-guide/index.html#installing-nvidia-drivers I also faced the same problem (nvcr.io/nvidia/tensorflow not working in WSL 2 after the update of Windows Insider version).
From your steps, I only had to re-install the CUDA driver ( https://developer.nvidia.com/cuda/wsl ). In my case I just had to run the installer like I normally did. I didn’t even have to restart WSL.
I ran the benchmark container to check if everything works:
docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmarkThe benchmark passed and my nvcr.io/nvidia/tensorflow container works again.
Works for me, bu the way i reinstall the Nvidia driver 460 everytime after an update
I have just updated to 20241 and
For me there is no problem with 20241. Best, Mickey
The nvidia docker works fine.
And so does Tensorflow
I’m using 20241, but cuda is still not working in mu wsl. Previously, when I use 20176, it worked
Yes now cuda on wsl is working in 20236. I am creating a restore point if this issue arises again in future.
I downgraded as part of recovery from 20226 back to 20221 without any problem. Everything worked as before the downgrade and wsl2 started working well.
Just to be on the safe side I made a full system backup beforehand.
If you have your pc for some time and “passed through” 20221 try several cycles or recovery to previous windows version.
If you never had it, e.g. your pc is newer than 2 weeks, you can install 20221 from the iso version keeping all your files. This is a safe procedure.
Nevertheless do not skip the backup. (Both Acronis and EaseUS Todo Backup work fine for me in making and restoring disk clones including all partitions).
You can get the 20221.1000 iso image in one of the links mentioned in previous messages.
Good luck, Mickey
It is working with 20221 but when testing the nvidia-docker-2 I get this error:
Can anyone advise of getting the nvidia container to run ? I followed these links: https://medium.com/@dalgibbard/docker-with-gpu-support-in-wsl2-ebbc94251cf5 https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
@strarsis In your case you need to use a Windows Insider build from the Dev Channel (build >=20150). CUDA in WSL2 won’t work in build 19041.
Same problem.
Rollback to previous version fixes it. For people who want to do it without reinstalling, go to Recovery > restore previous version of windows
Interesting, you seem to be missing the libdxcore libraries.