WSL: Running CUDA samples with multiple GPUs is failed
Version
Microsoft Windows [Version 10.0.22000.376]
WSL Version
- WSL 2
- WSL 1
Kernel Version
5.10.60.1
Distro Version
Ubuntu 20.04 and Ubuntu 18.04
Other Software
CPU: Intel® Core™ i9-9900X GPU: Nvidia Titan RTX * 4 (driver 510.06) RAM: 128GB
Repro Steps
Install CUDA on WSL
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
sudo apt-get update
sudo apt-get install -y cuda-toolkit-11-0
Run samples
cd /usr/local/cuda-11.0/samples/4_Finance/BlackScholes
sudo make
./BlackScholes
cd /usr/local/cuda-11.0/samples/1_Utilities/deviceQuery
sudo make
./deviceQuery
Expected Behavior
Return success
Actual Behavior
[./BlackScholes] - Starting...
CUDA error at ../../common/inc/helper_cuda.h:777 code=2(cudaErrorMemoryAllocation) "cudaGetDeviceCount(&device_count)"
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 2
-> out of memory
Result = FAIL
Diagnostic Logs
No response
About this issue
- Original URL
- State: open
- Created 5 years ago
- Comments: 18
I am also having this issue. I’m glad I found this thread, I thought I was going crazy. I’m using 4x A6000, WSL Ubuntu 20.04, CUDA 11.7.1. Any combination of
CUDA_VISIBLE_DEVICESworks except those that contain both device 1 and 2.I am having this issue now, almost exactly as described. Any combination of devices that don’t include both 1 and 2 work.
Running latest on WSL2, Ubuntu 22.04, CUDA 12.2. 4x RTX 6000 Ada. Similar to above, any combination of 1 and 3 fail. 0,1,2,3 fail 0,1,2 pass 0,1,3 fail 0,2,3 pass 1,2,3 fail 0,1 pass 0,2 pass 0,3 pass 1,2 pass 1,3 fail 2,3 pass
Same issue here, I’m using 4 2080TIs, WSL Ubuntu 20.04, kernel version 5.10.102.1-microsoft-standard-WSL2, CUDA 11.3. Combination of device 0 and 1 will result in “cudaGetDeviceCount returned 2 -> out of memory” failure. This problem has been discovered for nearly a year and is still not solved😥
Same issue here
@elsaco thanks for help. it works. However, it still fails when using GPUs 2 and 3 together. Is it normal?
CUDA_VISIBLE_DEVICES="{GPU}" ./deviceQueryCUDA_VISIBLE_DEVICES=0 ./deviceQueryCUDA_VISIBLE_DEVICES="0,1,2,3" ./deviceQuery@yes89929 you’re using multiple GPUs, therefore it might help troubleshooting by isolating a specific GPU.
Please try
CUDA_VISIBLE_DEVICES=0 ./deviceQueryto run the test on the first available GPU only.Also
CUDA_VISIBLE_DEVICES="0,1,2,3"to use all GPUs.