WSL: RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

I just follow the installation step by step here https://docs.nvidia.com/cuda/wsl-user-guide/index.html

I’m not trying to use docker I’m trying to use pytorch directly from python This is the code `` (base) gegg00@DESKTOP-KS0OLS9:~$ python Python 3.7.6 (default, Jan 8 2020, 19:59:22) [GCC 7.3.0] :: Anaconda, Inc. on linux Type “help”, “copyright”, “credits” or “license” for more information.

import torch torch.cuda.is_available() True torch.randn(5) tensor([-2.6408, -1.0831, -1.6984, 0.4742, -0.5909]) torch.randn(5).to(2) Traceback (most recent call last): File “<stdin>”, line 1, in <module> RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable ``

Environment

Platform ServicePack Version VersionString


Win32NT 10.0.20170.0 Microsoft Windows NT 10.0.20170.0

Ubuntu Release: 18.04

WSL 2: Linux version 4.19.121-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Fri Jun 19 21:06:10 UTC 2020

Steps to reproduce

I follow the steps https://docs.nvidia.com/cuda/wsl-user-guide/index.html Installed Anaconda Anaconda3-2020.02-Linux-x86_64.sh Installed pytorch conda install pytorch torchvision cudatoolkit=10.2 -c pytorch try the code `` (base) gegg00@DESKTOP-KS0OLS9:~$ python Python 3.7.6 (default, Jan 8 2020, 19:59:22) [GCC 7.3.0] :: Anaconda, Inc. on linux Type “help”, “copyright”, “credits” or “license” for more information.

import torch torch.cuda.is_available() True torch.randn(5) tensor([-2.6408, -1.0831, -1.6984, 0.4742, -0.5909]) torch.randn(5).to(2) Traceback (most recent call last): File “<stdin>”, line 1, in <module> RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable `` Getting this error: RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

Expected behavior

Actual behavior

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 8
  • Comments: 16 (1 by maintainers)

Most upvoted comments

I have the same problem and have the same kernel version but nvidia driver 460.20 torch.cuda.is_available() returns true but cannot perform any operation on cuda

I am having the same problem, with NVIDIA driver 460.20, and WSL 2 kernel version 4.19.128.

@yhtgrace Here is a quick fix:

Go to Settings -> Recovery -> Click on Get Started under Go back to previous version of Windows 10

This will take 5-10 minutes to roll back to previous version.

It may work. But there was an unfixed issue in version 20221 that caused the following error when running some Linux distros: “The remote procedure call failed”

I guess we just have to wait until Windows gets fixed again.

Thanks @rohitsingh02! Rolling back to 20221.1000 (while staying at WSL 2 4.19.128, NVIDIA 460.15) resolved the issue for me

FYI, this worked for me on build 20236! Both the minimal example (torch.randn(1).to(0)) and training a full torch model. Thanks for your help, everyone 😃

If it helps, what I chose to do as a workaround is simply run my CUDA code on the Windows command line as opposed to WSL. Of course, this requires that you reproduce whatever setup you had in WSL, e.g. installing packages. It’s not really a solution, because you’re just using a different OS, but hopefully it helps you do your work 😃

Unfortunately I was trying to use the fastai lib which specifically really doesn’t support Windows.

Build 20236 seems to work:

(fastai) forbiddenera@FORBIDDENERA:~$ sudo docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance.
        -fullscreen       (run n-body simulation in fullscreen mode)
        -fp64             (use double precision floating point values for simulation)
        -hostmem          (stores simulation data in host memory)
        -benchmark        (run benchmark to measure performance)
        -numbodies=<N>    (number of bodies (>= 1) to run in simulation)
        -device=<d>       (where d=0,1,2.... for the CUDA device to use)
        -numdevices=<i>   (where i=(number of CUDA devices > 0) to use for simulation)
        -compare          (compares simulation results running once on the default GPU and once on the CPU)
        -cpu              (run n-body simulation on the CPU)
        -tipsy=<file.bin> (load a tipsy model file for simulation)

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

> Windowed mode
> Simulation data stored in video memory
> Single precision floating point simulation
> 1 Devices used for simulation
MapSMtoCores for SM 7.5 is undefined.  Default to use 64 Cores/SM
GPU Device 0: "GeForce RTX 2070 SUPER" with compute capability 7.5

> Compute 7.5 CUDA device: [GeForce RTX 2070 SUPER]
40960 bodies, total time for 10 iterations: 57.387 ms
= 292.352 billion interactions per second
= 5847.040 single-precision GFLOP/s at 20 flops per interaction

Thanks guys! Now to just get fastai working…heh