WSL: WSL2 & CUDA does not work [v20226]

Environment

Windows build number: 10.0.20226.0
Your Distribution version: 18.04 / 20.04
Whether the issue is on WSL 2 and/or WSL 1: Linux version 4.19.128-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Tue Jun 23 12:58:10 UTC 2020

Steps to reproduce

Exactly followed instructions available at https://docs.nvidia.com/cuda/wsl-user-guide/index.html Tested on previously working Ubuntu WSL image (IIRC GPU last worked on 20206, than whole WSL2 stopped working) Tested also on newly created Ubuntu 18.04 and Ubuntu 20.04 images.

I have tested CUDA compatible NVIDIA drivers 455.41 & 460.20. I have tried removing all drivers etc. I have also tested using CUDA 10.2 & CUDA 11.0.

It was tested on two separate machines (one Intel + GTX1060, other Ryzen + RTX 2080Ti)

Issue tested directly in OS also in docker containers inside.

Example (directly in Ubuntu):

piotr@DESKTOP-FS6J3NT:/usr/local/cuda/samples/4_Finance/BlackScholes$ ./BlackScholes
[./BlackScholes] - Starting...
GPU Device 0: "Turing" with compute capability 7.5

Initializing data...
...allocating CPU memory for options.
...allocating GPU memory for options.
CUDA error at BlackScholes.cu:116 code=46(cudaErrorDevicesUnavailable) "cudaMalloc((void **)&d_CallResult, OPT_SZ)"

Example in container:

piotr@DESKTOP-FS6J3NT:/mnt/c/Users/pppnn$ docker run -it --gpus all -p 8888:8888 tensorflow/tensorflow:latest-gpu-py3-jupyter python
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2020-10-01 14:18:07.538627: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-10-01 14:18:07.624188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
>>> tf.test.is_gpu_available()
WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2020-10-01 14:18:32.359457: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-01 14:18:32.398949: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3200035000 Hz
2020-10-01 14:18:32.402692: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d06b70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-01 14:18:32.402748: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-01 14:18:32.409370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-10-01 14:18:32.877228: W tensorflow/compiler/xla/service/platform_util.cc:276] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2020-10-01 14:18:32.877370: I tensorflow/compiler/jit/xla_gpu_device.cc:136] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA
2020-10-01 14:18:32.879904: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:32.880192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1d:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.665GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-01 14:18:32.880277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-01 14:18:32.880340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-01 14:18:32.959947: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-01 14:18:32.973554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-01 14:18:33.111736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-01 14:18:33.127902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-01 14:18:33.128018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-01 14:18:33.128535: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:33.129170: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:33.129403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-10-01 14:18:33.131671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/test_util.py", line 1513, in is_gpu_available
    for local_device in device_lib.list_local_devices():
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/device_lib.py", line 43, in list_local_devices
    _convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable
>>>
>>>
>>>
>>>
>>> tf.config.list_physical_devices('GPU')
2020-10-01 14:18:55.610151: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:55.610510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1d:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.665GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-01 14:18:55.610579: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-01 14:18:55.610623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-01 14:18:55.610676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-01 14:18:55.610719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-01 14:18:55.610762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-01 14:18:55.610805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-01 14:18:55.610846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-01 14:18:55.611251: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:55.611765: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:18:55.611999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
>>>
>>>
>>>
>>> tf.test.gpu_device_name()
2020-10-01 14:20:08.762060: W tensorflow/compiler/xla/service/platform_util.cc:276] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2020-10-01 14:20:08.762222: I tensorflow/compiler/jit/xla_gpu_device.cc:136] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA
2020-10-01 14:20:08.762863: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:20:08.763201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1d:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.665GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-01 14:20:08.763263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-01 14:20:08.763316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-01 14:20:08.763358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-01 14:20:08.763379: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-01 14:20:08.763428: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-01 14:20:08.763480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-01 14:20:08.763533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-01 14:20:08.763898: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:20:08.764536: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:967] could not open file to read NUMA node: /sys/bus/pci/devices/0000:1d:00.0/numa_node
Your kernel may have been built without NUMA support.
2020-10-01 14:20:08.764810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/test_util.py", line 112, in gpu_device_name
    for x in device_lib.list_local_devices():
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/device_lib.py", line 43, in list_local_devices
    _convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable
>>>

Expected behavior

CUDA working inside WSL2

Actual behavior

All tests which are using CUDA inside WSL Ubuntu are resulting with various CUDA errors - mostly referring to no CUDA devices available.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 75
  • Comments: 176 (5 by maintainers)

Most upvoted comments

We have identified the issue and are working on a fix.

Build 20236 blog post annonuncement

We fixed a regression that was breaking NVIDIA CUDA vGPU acceleration in the Windows Subsystem for Linux. Please see this GitHub thread for full details.

Can confirm that this is currently still happening in the most recent version – 20231

I confirm the same issue exists in 20231.1000

I am having the same problem. WIndows 10 build 20226 and Nvidia driver 460.20. It is great to see that it is not just my install. I hope that this can be fixed soon.

And now I can also confirm that it will work if you roll back to the previous build 20221. You can download the (old) iso file from Microsoft and re-install without losing any data.

Windows insider updated to 20262.1 With the update, the regular nvidia driver was re-installed. Not good for wsl2. Also, nvidia issued a new driver that you should install in windows: https://developer.nvidia.com/cuda/wsl The new drive is 465.12 Download and install it. You must reboot after the install or wsl2 will not see the gpu as of yet. Start wsl2

$ python
> import tensorflow as tf
>  tf.test.is_gpu_available()
2020-11-19 17:15:13.395676: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
True

Then do the above docker procedures to check that the docker aslo works. In my case, it works fine.

After doing this, it still didn’t work for me.

I was able to get it to work by following it up with this:

  • Installed everything else that Windows Update wanted to install and rebooted
  • Reinstalled the 465.12 driver
    • On the first screen, I selected “Drivers + GeForce Experience”
    • On the second screen, I selected Custom, then pressed Next and checked “Perform a clean install”
  • Uninstalled all my existing Ubuntu installations
  • Restarted Windows
  • Installed Ubuntu from Microsoft store (the one called “Ubuntu” with no version) (it installed Ubuntu 20.04)
  • Restarted Windows
  • Followed these instructions to add the sources to my sources lists, but I modified the URLs to be 2004 instead of 1804:
    • sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
    • sudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
    • sudo apt-get update
  • Continued to follow the instructions by installing Nvidia Toolkit 11.1 (not 11.0)
    • sudo apt-get install -y cuda-toolkit-11-1
  • Followed these instructions to test the installation
    • wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    • bash Miniconda3-latest-Linux-x86_64.sh
      • When it asks whether to init conda, hit yes
    • restart terminal
    • conda config --set auto_activate_base false
    • restart terminal
    • conda create --name directml python=3.6
    • conda activate directml
    • pip install tensorflow-directml
  • Then in Python
    • import tensorflow.compat.v1 as tf
    • tf.enable_eager_execution(tf.ConfigProto(log_device_placement=True))
    • print(tf.add([1.0, 2.0], [3.0, 4.0]))

And there, I finally saw the name of my GPU pop up!

After that I went back and tried the ./BlackScholes test:

  • cd /usr/local/cuda/samples/4_Finance/BlackScholes
  • sudo make
  • ./BlackScholes

And it also worked!

Seems this issue has been tagged “fix inbound” for quite a while, but is not even mentioned in the 20231 release note(not even in the known issues part).

Could more attention be paid to this issue? This is the main blocker for me as a CUDA programmer.

It is also fixed for me in 20236 so I am closing the issue. Thanks to all commenters and also WSL developers for fixing the issue.

Cuda on WSL2 works perfectly on build 20236. The problem seems to be resolved.

I am having the same issue. Everything was working flawlessly this morning but then I have updated to 20226.1000 from 20221.1000 and it does not work anymore (tried reinstalling nvidia drivers, etc.) with error that all cuda devices are busy or unavailable.

Edit: After going back to version 20221, everything works again, thus it confirms that the new version caused the problem.

可以共享c:\ Windows \ System32 \ lxss \ lib的内容吗?

lib_list

@ccs96307 Now what can we say phew… solving 1 problem leads to another

With Windows 10 (non-insider) 2021H2 November update, CUDA now works in WSL 2.

Same issue here on the 20231 version , this issue needs to be fixed soon!

Error on WSL2

Environment: Windows insider program 21332.1000,  NVIDIA Driver 461.72 on Windows (for GeForce GTX 1660 Ti) , WSL2  Ubuntu 20-04, Error: CUDA error at …/…/common/inc/helper_cuda.h:779 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

After installing and having the NVIDIA driver working on Windows. I followed the NVIDA site https://docs.nvidia.com/cuda/wsl-user-guide/index.html instructions  (adapted for Ubuntu 20-04 because the example uses 18-04)

Steps on Ubuntu-20.04:

apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub sudo sh -c ‘echo “deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /” > /etc/apt/sources.list.d/cuda.list’ sudo apt-get install -y cuda-toolkit-11-2

After compiling and executing the ./BlackScholes example I get (again) the same error described on this thread. Although there is a comment earlier that it worked again for user EtienneT with the Windows insider release 21332.1000, it didn’t work for me. It does the same as with the previous insider release 21327.1000. Note that Microsoft release notes for 21327 admits than GPU computing won’t work (a regression bug), but in the release notes for 21332 MS says it is fixed. I get the same error on both Windows insider releases.

Situation on Windows 10

However the GPU works on Windows with using Pytorch

Pytorch Windows, Installed with: conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Code:  print(torch. cuda. get_device_name(torch. cuda. current_device())) It prints: GeForce GTX 1660 Ti with Max-Q Design

But tensorflow 2.4.1 on Windows installed with: pip install tensorflow It recognizes the GPU but fails downstream with a known problem with Tensorflow for which I cannot find updated info. Code: if tf.test.gpu_device_name() != ‘/device:GPU:0’:   print(‘WARNING: GPU device not found.’) else:   print(‘SUCCESS: Found GPU: {}’.format(tf.test.gpu_device_name()))   physical_devices = tf.config.list_physical_devices(‘GPU’)   tf.config.experimental.set_memory_growth(physical_devices[0], True) #thia should solve the cuda solver problem

It prints ‘SUCCESS: Found GPU:’ etc, but later in the process it has a problem in one of the cuda libraries. Here is the output of the run:

021-03-11 21:06:25.144177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4744 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti with Max-Q Design, pci bus id: 0000:02:00.0, compute capability: 7.5) 2021-03-11 21:06:26.697412: F tensorflow/core/util/cuda_solvers.cc:115] Check failed: cusolverDnCreate(&cusolver_dn_handle) == CUSOLVER_STATUS_SUCCESS Failed to create cuSolverDN instance

Note that for Tensorflow I had to download the CUDA Toolk kit 11.2 and associated Cudadnn libraries.

Any help will be appreciated. Petrarca Bruto

Windows insider updated to 20262.1 With the update, the regular nvidia driver was re-installed. Not good for wsl2. Also, nvidia issued a new driver that you should install in windows: https://developer.nvidia.com/cuda/wsl The new drive is 465.12 Download and install it. You must reboot after the install or wsl2 will not see the gpu as of yet. Start wsl2

$ python
> import tensorflow as tf
>  tf.test.is_gpu_available()
2020-11-19 17:15:13.395676: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
True

Then do the above docker procedures to check that the docker aslo works. In my case, it works fine.

Build 20236 fixed it for me 👍

Same problem on 20231. Cuda samples error out, for example matrixMul:

[Matrix Multiply Using CUDA] - Starting...
GPU Device 0: "Turing" with compute capability 7.5

MatrixA(320,320), MatrixB(640,320)
CUDA error at matrixMul.cu:130 code=46(cudaErrorDevicesUnavailable) "cudaMallocHost(&h_A, mem_size_A)"

I hope this is fixed soon enough

@mitchellvitez I really don’t think going back to 20201 is safe move considering previous bugs in 20201 as well

I have the error remote procedure call failed in the last version, and I have this issue after upgrade. So…when I recovery does it mean I will get the remote procedure call failed back 😿

I have the same problem in 20226. My build also contains same 8 files in lxss\lib. But I get cudaErrorDevicesUnavailable.

Is there a way to roll back 20221? Using “Go back to previous version of Windows 10” sends me to 19041.508.

Yes, you can install 20221 from https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewadvanced

It’s three days I continue trying different solutions but seems there’s none apart from downgrading… I get the same error as the above posts either with cuda-toolkit-11-0 and cuda toolkit-11-1 with my small nvidia GeForce MX150. The error constantly thrown up is (for the .BlackScholes example given in the CUDA on WSL guide https://docs.nvidia.com/cuda/wsl-user-guide/index.html) is: CUDA error at BlackScholes.cu:116 code=46(cudaErrorDevicesUnavailable) "cudaMalloc((void **)&d_CallResult, OPT_SZ)" I bookmarked this page and following updates daily!

I have the same problem in 20226.

  • WSL2 Ubuntu 18.04
  • Kerver Version 4.19.128
  • driver version 460.20
  • RTX 2060
  • Windows insider 20226 https://aka.ms/AA9utty

Cuda device query is ok.

./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: “GeForce RTX 2060” CUDA Driver Version / Runtime Version 11.2 / 10.0 CUDA Capability Major/Minor version number: 7.5 Total amount of global memory: 6144 MBytes (6442450944 bytes) (30) Multiprocessors, ( 64) CUDA Cores/MP: 1920 CUDA Cores GPU Max Clock rate: 1200 MHz (1.20 GHz) Memory Clock rate: 7001 Mhz Memory Bus Width: 192-bit L2 Cache Size: 3145728 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 1024 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: Yes Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): Yes Device supports Compute Preemption: Yes Supports Cooperative Kernel Launch: Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.2, CUDA Runtime Version = 10.0, NumDevs = 1 Result = PASS

But cuda utility does not worked.

[./BlackScholes] - Starting… GPU Device 0: “GeForce RTX 2060” with compute capability 7.5

Initializing data… …allocating CPU memory for options. …allocating GPU memory for options. CUDA error at BlackScholes.cu:116 code=46(cudaErrorDevicesUnavailable) “cudaMalloc((void **)&d_CallResult, OPT_SZ)”

Below is strace log. BlackScholes_cuda_error_log.zip

Oh too bad, I also encountered this problem. I was so happy when wsl worked again in the 20226 version, but cuda couldn’t work. I was left out of the cold. I tried the following solutions, but none of them worked for me.

  1. Reinstall the graphics card driver 460.20.

  2. Recompile cuda dependent environment library.

  3. Uninstall wsl2 and kernel program and reinstall.

I just updated to 20262.1010 and reinstalled nvidia driver 465.12, and installed Ubuntu 20.04 WSL2, and setup accordingly (used the CUDA WSL2 tutorial and replaced 1804 with 2004), and it just worked ! and even my previous Ubuntu 18.04 started working for CUDA (it wasn’t working on older build), Thanks to @serg06 !

I downgraded the version to 20221, the WSL 2 I installed showed the error message “remote procedure call failed”.

Same issue here. If you use the Build 20231, I don’t suggest to downgrade because it introduces a new error about user accounts. If you want to try, please consider backup.

@kivancguckiran I just joined the insider build so I’m in the same boat. It would probably take like 4 hours, but you could probably revert windows to the previous version (non-insider) maybe then go specifically to 20221. I’m not going to try it and just wait though.

@kerim371 I’m afraid it won’t work.

Note: CUDA on WSL 2 is enabled on GPUs starting with the Kepler architecture

https://docs.nvidia.com/cuda/wsl-user-guide/index.html

What could I do if I Nvidia driver manager can’t find appropriate driver for me? Samsung RC720 notebook OS build: 21370.1 Basic GPU is Intel, additional Nvidia GeForce GT 520M According to Nvidia docs my GeForce GT 520M supports CUDA image

Windows insider updated to 21327.1000 cuda broken again. previous insider version work well.

Error message with BlackScholes: CUDA error at ../../common/inc/helper_cuda.h:779 code=35(cudaErrorInsufficientDriver) "cudaGetDeviceCount(&device_count)

After re-install(Custom-Perform a clean install-restart pc) the CUDA driver (wsl ready) all goes well again.

This makes it work. Basically install docker on wsl2. https://dev.to/bartr/install-docker-on-windows-subsystem-for-linux-v2-ubuntu-5dl7

After restart of wsl2 do the following procedure:

$ docker ps
$ docker ps -aq
$ docker stop $(docker ps -aq)
$ docker stop # dockers reported by ps
$ rm ~/.docker/config.json
$ sudo service docker stop
$ sudo service docker start
$ sudo mkdir /sys/fs/cgroup/systemd
$ sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
$ docker ps
$ docker run hello-world
$ docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
GPU Device 0: "GeForce RTX 2080 Ti" with compute capability 7.5

> Compute 7.5 CUDA device: [GeForce RTX 2080 Ti]
69632 bodies, total time for 10 iterations: 117.583 ms
= 412.357 billion interactions per second
= 8247.147 single-precision GFLOP/s at 20 flops per interaction

One more thing you need to know. The usual docker-compose (last one is version 1.27) that gets installed with the docker does not support starting dockers with gpu. The only way to manage docker containers with gpu is using an experimental version of docker-compose that does support management of docker containers that use a gpu.

$ which docker-compose
$ sudo rm rf <folder of docker compose> 
$ pip install git+https://github.com/yoanisgil/compose.git@device-requests
$ docker-compose --version
docker-compose version 1.26.0dev, build unknown
# to use it you have to make sure that API version is 1.40. This can be done on the same command-line:
$ COMPOSE_API_VERSION=1.40 docker-compose up
    NVIDIA_DRIVER_PRESENT: "True"
    container_name: docker_name_needing_gpu
    device_requests:
        - capabilities:
            - "gpu"

I confirm the same issue exists in 20231.1000

I confirm the issue is resolved in 20236.1000. Thanks to all who contributed momentum toward this.

Same issue for me on Build 20231, WSL2 and GTX 1650:

root@LAPTOP:/usr/local/cuda/samples/4_Finance/BlackScholes# ./BlackScholes
[./BlackScholes] - Starting...
GPU Device 0: "Turing" with compute capability 7.5

Initializing data...
...allocating CPU memory for options.
...allocating GPU memory for options.
CUDA error at BlackScholes.cu:116 code=46(cudaErrorDevicesUnavailable) "cudaMalloc((void **)&d_CallResult, OPT_SZ)"

No GPU is listed after issuing the command lspci on Ubuntu 18.04:

978b:00:00.0 3D controller: Microsoft Corporation Device 008e
ab50:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
abff:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)
ae20:00:00.0 3D controller: Microsoft Corporation Device 008e
bf2d:00:00.0 SCSI storage controller: Red Hat, Inc. Virtio filesystem (rev 01)

The Tensorflow function list_local_devices returns:

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 13820027289368611552
, name: "/device:XLA_CPU:0"
device_type: "XLA_CPU"
memory_limit: 17179869184
locality {
}
incarnation: 11825873724132199309
physical_device_desc: "device: XLA_CPU device"
]

Build 20201 should be a good build stop too. CUDA in WSL2 works since build 20145.

@FluorineDog sweet summer child, getting an MS response at all is an anomaly, and 3 days old is milliseconds (been waiting on #4150 for 1yr). I anticipate staying in this rolled back build for 3-6mo. See this for disabling auto update/restart (though not sure if Dev Channel will auto update if you’ve rolled back). Anyway, definitely do the rollback soon, and don’t twiddle your thumbs on a fix.

Previous builds in the near past trigger another fatal bug, while ancient ones don’t even deliver the needed functionality. That’s why I’m checking this thread constantly. Once I can run my first CUDA program, I’ll freeze my dev channel gladly, but sadly not now.

I hope so, @onomatopellan !

I froze my automatic update for 7 days. NVIDIA just released driver 456.71 I keep my 460.20.

By the way, Nvidia’s “default” install is CUDA 11.1 and cuDNN 8.0 most software requires earlier versions that do work perfectly well.

For example, there is no mxnet for cuda 11.x.

Here is how you install a previous version: Start here: (do not fall into the trap of installation for Ubuntu. Crack the cryptic installation for WSL !) https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#wsl-x86_64-rpm

import tensorflow as tf tf.test.is_gpu_available()

Note: if for some reason your apt-geu update/upgade/dist-upgrade cause cuda 11.x to be installed over your previous version, simply run the three dpkg -i on cuda, cudnn and cudnn-dev and your version will be back on line. Usually nothing eles is needed.

The following worked for me:

  • Re-install the 465.42 WSL driver (https://developer.nvidia.com/cuda/wsl)
  • Disable and enable the GPU driver from device manager. This a known issue by NVIDIA as described in their documentation:https://docs.nvidia.com/cuda/wsl-user-guide/index.html “Note:NVIDIA is aware of a specific installation issue reported on mobile platforms with the WIP driver 465.12 posted on 11/16/2020. A known workaround will be to disable and reenable the GPU adapter from device manager at system start. We are working on a fix for this issue and will have an updated driver soon.As an alternative, users may opt to roll back to an earlier driver from device manager driver updates.”

Windows insider updated to 21327.1000 cuda broken again. previous insider version work well.

This is called out in the flight notes: https://blogs.windows.com/windows-insider/2021/03/03/announcing-windows-10-insider-preview-build-21327/

Windows Subsystem for Linux (WSL) users who upgrade to this build will be unable to use the GPU Compute feature. We’re working on a fix for this. Users who do a clean install will not be affected.

Updated to Insider version 20270.1 still with 465.12 and all is working well as before.

I can confirm that installing the latest docker edge, Microsoft Edge packages, and NVIDIA 465.12 makes it all work again.

I had stopped working on this for a month because an update killed it,it was badly broken.

Fortunately it is working again.

Even better than before actually because now NVIDIA DOCKER can run in multiple containers simultaneously!!!

Windows insider updated to 20262.1 With the update, the regular nvidia driver was re-installed. Not good for wsl2. Also, nvidia issued a new driver that you should install in windows: https://developer.nvidia.com/cuda/wsl The new drive is 465.12 Download and install it. You must reboot after the install or wsl2 will not see the gpu as of yet. Start wsl2

$ python
> import tensorflow as tf
>  tf.test.is_gpu_available()
2020-11-19 17:15:13.395676: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
True

Then do the above docker procedures to check that the docker aslo works. In my case, it works fine.

After doing this, it still didn’t work for me.

I was able to get it to work by following it up with this:

  • Installed everything else that Windows Update wanted to install and rebooted

  • Reinstalled the 465.12 driver

    • On the first screen, I selected “Drivers + GeForce Experience”
    • On the second screen, I selected Custom, then pressed Next and checked “Perform a clean install”
  • Uninstalled all my existing Ubuntu installations

  • Restarted Windows

  • Installed Ubuntu from Microsoft store (the one called “Ubuntu” with no version) (it installed Ubuntu 20.04)

  • Restarted Windows

  • Followed these instructions to add the sources to my sources lists, but I modified the URLs to be 2004 instead of 1804:

    • sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
    • sudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
    • sudo apt-get update
  • Continued to follow the instructions by installing Nvidia Toolkit 11.1 (not 11.0)

    • sudo apt-get install -y cuda-toolkit-11-1
  • Followed these instructions to test the installation

    • wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

    • bash Miniconda3-latest-Linux-x86_64.sh

      • When it asks whether to init conda, hit yes
    • restart terminal

    • conda config --set auto_activate_base false

    • restart terminal

    • conda create --name directml python=3.6

    • conda activate directml

    • pip install tensorflow-directml

  • Then in Python

    • import tensorflow.compat.v1 as tf
    • tf.enable_eager_execution(tf.ConfigProto(log_device_placement=True))
    • print(tf.add([1.0, 2.0], [3.0, 4.0]))

And there, I finally saw the name of my GPU pop up!

After that I went back and tried the ./BlackScholes test:

  • cd /usr/local/cuda/samples/4_Finance/BlackScholes
  • sudo make
  • ./BlackScholes

And it also worked!

Thanks for posting your steps. I also run Ubuntu in WSL2 and use the Nvidia docker containers as explained in https://docs.nvidia.com/cuda/wsl-user-guide/index.html#installing-nvidia-drivers I also faced the same problem (nvcr.io/nvidia/tensorflow not working in WSL 2 after the update of Windows Insider version).

From your steps, I only had to re-install the CUDA driver ( https://developer.nvidia.com/cuda/wsl ). In my case I just had to run the installer like I normally did. I didn’t even have to restart WSL.

I ran the benchmark container to check if everything works:

docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

The benchmark passed and my nvcr.io/nvidia/tensorflow container works again.

Works for me, bu the way i reinstall the Nvidia driver 460 everytime after an update

I have just updated to 20241 and

  1. nvidia docker works fine
  2. tensorflow report my GPU 2080ti, as true.
  3. tensorflow reports device name correctly. I suggest you check that maybe your cuda version was updated and no longer matches your TF version. If so, scroll back - there is a procedure how to install any previous cuda version.

For me there is no problem with 20241. Best, Mickey

$ docker ps
$ docker stop # dockers reported by ps
$ rm ~/.docker/config.json
$ sudo service docker stop
$ sudo service docker start
$ sudo mkdir /sys/fs/cgroup/systemd
$ sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
$ docker ps
$ docker run hello-world
$ docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
> Windowed mode
> Simulation data stored in video memory
> Single precision floating point simulation
> 1 Devices used for simulation
MapSMtoCores for SM 7.5 is undefined.  Default to use 64 Cores/SM
GPU Device 0: "GeForce RTX 2080 Ti" with compute capability 7.5

> Compute 7.5 CUDA device: [GeForce RTX 2080 Ti]
69632 bodies, total time for 10 iterations: 118.318 ms
= 409.797 billion interactions per second
= 8195.939 single-precision GFLOP/s at 20 flops per interaction

The nvidia docker works fine.

And so does Tensorflow

$ python
> import tensorflow as tf
> tf.test.is_gpu_available()
Your kernel may have been built without NUMA support.
2020-10-25 00:21:10.409370: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555df5500de0 executing computations on platform CUDA. Devices:
2020-10-25 00:21:10.409404: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
True
> tf.test.gpu_device_name()
2020-10-25 00:22:14.701525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 9630 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:09:00.0, compute capability: 7.5)
'/device:GPU:0'

I’m using 20241, but cuda is still not working in mu wsl. Previously, when I use 20176, it worked

import torch torch.cuda.current_device() Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “/home/xiaothan/miniconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/cuda/init.py”, line 377, in current_device _lazy_init() File “/home/xiaothan/miniconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/cuda/init.py”, line 196, in _lazy_init _check_driver() File “/home/xiaothan/miniconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/cuda/init.py”, line 101, in _check_driver http://www.nvidia.com/Download/index.aspx"“”) AssertionError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx_

Yes now cuda on wsl is working in 20236. I am creating a restore point if this issue arises again in future.

I downgraded as part of recovery from 20226 back to 20221 without any problem. Everything worked as before the downgrade and wsl2 started working well.

Just to be on the safe side I made a full system backup beforehand.

If you have your pc for some time and “passed through” 20221 try several cycles or recovery to previous windows version.

If you never had it, e.g. your pc is newer than 2 weeks, you can install 20221 from the iso version keeping all your files. This is a safe procedure.

Nevertheless do not skip the backup. (Both Acronis and EaseUS Todo Backup work fine for me in making and restoring disk clones including all partitions).

You can get the 20221.1000 iso image in one of the links mentioned in previous messages.

Good luck, Mickey

It is working with 20221 but when testing the nvidia-docker-2 I get this error:

docker: Error response from daemon: could not select device driver “” with capabilities: [[gpu]].
ERRO[0084] error waiting for container: context canceled

Can anyone advise of getting the nvidia container to run ? I followed these links: https://medium.com/@dalgibbard/docker-with-gpu-support-in-wsl2-ebbc94251cf5 https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

  • Windows 10 Version 2004 (Build 19041.546)

@strarsis In your case you need to use a Windows Insider build from the Dev Channel (build >=20150). CUDA in WSL2 won’t work in build 19041.

Same problem.

  • WSL2 Ubuntu 20.04
  • driver version 460.20
  • Razer blade advanced 4K 2019
  • RTX 2080 Max Q
  • Windows insider 20226

Rollback to previous version fixes it. For people who want to do it without reinstalling, go to Recovery > restore previous version of windows

Interesting, you seem to be missing the libdxcore libraries.