WSL: Running CUDA samples with multiple GPUs is failed

Version

Microsoft Windows [Version 10.0.22000.376]

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

5.10.60.1

Distro Version

Ubuntu 20.04 and Ubuntu 18.04

Other Software

CPU: Intel® Core™ i9-9900X GPU: Nvidia Titan RTX * 4 (driver 510.06) RAM: 128GB

Repro Steps

Install CUDA on WSL

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub

sudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'

sudo apt-get update

sudo apt-get install -y cuda-toolkit-11-0

Run samples

cd /usr/local/cuda-11.0/samples/4_Finance/BlackScholes

sudo make

./BlackScholes
cd /usr/local/cuda-11.0/samples/1_Utilities/deviceQuery

sudo make

./deviceQuery

Expected Behavior

Return success

Actual Behavior

[./BlackScholes] - Starting...
CUDA error at ../../common/inc/helper_cuda.h:777 code=2(cudaErrorMemoryAllocation) "cudaGetDeviceCount(&device_count)"
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 2
-> out of memory
Result = FAIL

Diagnostic Logs

No response

About this issue

Most upvoted comments

I am also having this issue. I’m glad I found this thread, I thought I was going crazy. I’m using 4x A6000, WSL Ubuntu 20.04, CUDA 11.7.1. Any combination of CUDA_VISIBLE_DEVICES works except those that contain both device 1 and 2.

I am having this issue now, almost exactly as described. Any combination of devices that don’t include both 1 and 2 work.

Running latest on WSL2, Ubuntu 22.04, CUDA 12.2. 4x RTX 6000 Ada. Similar to above, any combination of 1 and 3 fail. 0,1,2,3 fail 0,1,2 pass 0,1,3 fail 0,2,3 pass 1,2,3 fail 0,1 pass 0,2 pass 0,3 pass 1,2 pass 1,3 fail 2,3 pass

Same issue here, I’m using 4 2080TIs, WSL Ubuntu 20.04, kernel version 5.10.102.1-microsoft-standard-WSL2, CUDA 11.3. Combination of device 0 and 1 will result in “cudaGetDeviceCount returned 2 -> out of memory” failure. This problem has been discovered for nearly a year and is still not solved😥

Same issue here

@elsaco thanks for help. it works. However, it still fails when using GPUs 2 and 3 together. Is it normal?

CUDA_VISIBLE_DEVICES="{GPU}" ./deviceQuery

GPU Result
0 PASS
1 PASS
2 PASS
3 PASS
0,1 PASS
0,1,2 PASS
0,1,2,3 FAIL
0,3 PASS
1,3 PASS
2,3 FAIL
0,1,3 PASS
1,2,3 FAIL

CUDA_VISIBLE_DEVICES=0 ./deviceQuery

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA TITAN RTX"
  CUDA Driver Version / Runtime Version          11.6 / 11.0
  CUDA Capability Major/Minor version number:    7.5
  Total amount of global memory:                 24576 MBytes (25769476096 bytes)

CUDA_VISIBLE_DEVICES="0,1,2,3" ./deviceQuery

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 2
-> out of memory
Result = FAIL

@yes89929 you’re using multiple GPUs, therefore it might help troubleshooting by isolating a specific GPU.

Please try CUDA_VISIBLE_DEVICES=0 ./deviceQuery to run the test on the first available GPU only.

Also CUDA_VISIBLE_DEVICES="0,1,2,3" to use all GPUs.