WSL: In WSL2's docker container : GPU access blocked by the operating system
Windows Version
Microsoft Windows [Version 10.0.22621.1555]
WSL Version
1.2.0.0
Are you using WSL 1 or WSL 2?
- WSL 2
- WSL 1
Kernel Version
5.15.90.1
Distro Version
Ubuntu-20.04
Other Software
-
Docker version 23.0.3, build 3e7cbfd
-
NVIDIA Container Runtime version 1.13.0 commit: b7079454b5b8fed1390ce78ca5a3343748f62657 spec: 1.0.2-dev runc version 1.1.5 commit: v1.1.5-0-gf19387a spec: 1.0.2-dev go: go1.19.7 libseccomp: 2.5.1
-
NVIDIA GeForce Game Ready driver version : 531.41 (installed in windows11)
Repro Steps
nvidia-smi works well in wsl2, but it doesn’t work properly in the docker container started in wsl2, with error “Failed to initialize NVML: GPU access blocked by the operating system”. I use the official image provided by Pytorch and am confident that Docker-ce and nvidia_container_toolkit has been installed correctly. In fact, when I use the same installation script in the Ubuntu system, the GPU in the container works well. Here is the command i install docker-ce and nvidia-container-toolkit.
- Install docker-ce in wsl2:
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
# here use aliyun mirror.
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install -y docker-ce
service docker start
- install nvidia-container-toolkit in wsl2:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
service docker restart
- pull nvidia-gpu-supported image and try to use GPU in container
docker run --rm --runtime nvidia --gpus all pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime nvidia-smi
Expected Behavior

Actual Behavior

Diagnostic Logs
No response
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 7
- Comments: 48 (7 by maintainers)
Inside the file /etc/nvidia-container-runtime/config.toml change no-cgroups from true to false
Yep downgrading fixed it. Thank you very much @jucysoft ❤️ ❤️ ❤️
Detectron2 finally is working again.
I can also view nvidia-smi inside the docker container:
I am having the same issues, anyone else got this working on WSL 2.x ?
Even though running the
nvidia-smicommand works fine, outputs all the correct GPU information:Hope someone is able to help me out, thanks a lot!
Edit: even after downgrading WSL to
1.2.3.0(even tried to downgrade to1.1.6.0), the issue persists for me:Just tested @lpdink steps on v1.3.10 and it worked:
Setup:
.wslconfigwithkernelCommandLine[boot] commandto umount the remainingcgroupmounts and mount/sys/fs/cgroupascgroup2I also tested Docker Desktop based on their doc: https://www.docker.com/blog/wsl-2-gpu-support-for-docker-desktop-on-nvidia-gpus/ Same results, all good too
Hope this helps and hopefully more people will be able to confirm it works now too (so we can get back to cgroup2 😇)
This issue occurs after updating wsl from 1.1.6 to 1.2.0.
This worked for me, thanks! I actually was even able to remove
kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1from .wslconfigOh, I have been obsessed by this very bug for couple of days! Thanks you guys for letting me know what the problem is! I have tried wsl 1.2.3 just now and It works very well! 😃
@l1377687647 I noticed that the latest version 1.2.3 has fixed this issue, but it has not been pushed to the Microsoft Store yet, you can download it in release and install it in PowerShell (admin mode) with the Add-AppxPackage command. You may need to call wsl --shutdown at first and terminate the wsl process via Task Manager. Or downgrade to 1.1.6 as above, please note that 1.1.6 is still the version of wsl2 instead of wsl1.
Yeah, very cool! Do you mind telling me how to downgrade to a previous Version of WSL (1.1.6) please? I have not been able to find that out yet.
@maurange well, just download 1.2.3 or 1.2.4 .msixbundle in release. Open powershell with admain mode:
if nothing wrong, this will work:)
I am using wsl 1.2.5 and facing this issue. Is wsl 1.2.5 broke GPU driver again?
Same error here, did you fixed your problem after downgrading it?
Its fine. Just figured it out. Just go to the GitRepo and download the msixbundle from the previous version and just double click it.