WSL: In WSL2's docker container : GPU access blocked by the operating system

Windows Version

Microsoft Windows [Version 10.0.22621.1555]

WSL Version

1.2.0.0

Are you using WSL 1 or WSL 2?

  • WSL 2
  • WSL 1

Kernel Version

5.15.90.1

Distro Version

Ubuntu-20.04

Other Software

  • Docker version 23.0.3, build 3e7cbfd

  • NVIDIA Container Runtime version 1.13.0 commit: b7079454b5b8fed1390ce78ca5a3343748f62657 spec: 1.0.2-dev runc version 1.1.5 commit: v1.1.5-0-gf19387a spec: 1.0.2-dev go: go1.19.7 libseccomp: 2.5.1

  • NVIDIA GeForce Game Ready driver version : 531.41 (installed in windows11)

Repro Steps

nvidia-smi works well in wsl2, but it doesn’t work properly in the docker container started in wsl2, with error “Failed to initialize NVML: GPU access blocked by the operating system”. I use the official image provided by Pytorch and am confident that Docker-ce and nvidia_container_toolkit has been installed correctly. In fact, when I use the same installation script in the Ubuntu system, the GPU in the container works well. Here is the command i install docker-ce and nvidia-container-toolkit.

  • Install docker-ce in wsl2:
apt-get update

apt-get install -y apt-transport-https ca-certificates curl software-properties-common 

# here use aliyun mirror.
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install -y docker-ce

service docker start
  • install nvidia-container-toolkit in wsl2:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt-get update
apt-get install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
service docker restart
  • pull nvidia-gpu-supported image and try to use GPU in container
docker run --rm --runtime nvidia --gpus all pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime nvidia-smi

Expected Behavior

image

Actual Behavior

image

Diagnostic Logs

No response

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 7
  • Comments: 48 (7 by maintainers)

Most upvoted comments

I am having the same issues, anyone else got this working on WSL 2.x ?

> wsl --version
WSL version: 2.0.14.0
Kernel version: 5.15.133.1-1
> docker run --rm --runtime nvidia --gpus all pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Even though running the nvidia-smi command works fine, outputs all the correct GPU information:

 nvidia-smi
Tue Mar  5 13:33:31 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.10              Driver Version: 551.61         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
...

Hope someone is able to help me out, thanks a lot!

Edit: even after downgrading WSL to 1.2.3.0 (even tried to downgrade to 1.1.6.0), the issue persists for me:

> wsl --version
WSL version: 1.2.3.0
Kernel version: 5.15.90.1
> docker run --rm --gpus all ubuntu nvidia-smi

Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Inside the file /etc/nvidia-container-runtime/config.toml change no-cgroups from true to false

wsl --version
WSL version: 2.1.5.0
Kernel version: 5.15.146.1-2

Yep downgrading fixed it. Thank you very much @jucysoft ❤️ ❤️ ❤️

Detectron2 finally is working again.

I can also view nvidia-smi inside the docker container: grafik

I am having the same issues, anyone else got this working on WSL 2.x ?

> wsl --version
WSL version: 2.0.14.0
Kernel version: 5.15.133.1-1
> docker run --rm --runtime nvidia --gpus all pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Even though running the nvidia-smi command works fine, outputs all the correct GPU information:

 nvidia-smi
Tue Mar  5 13:33:31 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.10              Driver Version: 551.61         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
...

Hope someone is able to help me out, thanks a lot!

Edit: even after downgrading WSL to 1.2.3.0 (even tried to downgrade to 1.1.6.0), the issue persists for me:

> wsl --version
WSL version: 1.2.3.0
Kernel version: 5.15.90.1
> docker run --rm --gpus all ubuntu nvidia-smi

Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Just tested @lpdink steps on v1.3.10 and it worked: image

Setup:

  • enable cgroupv2 on .wslconfig with kernelCommandLine
  • on Ubuntu: add a [boot] command to umount the remaining cgroup mounts and mount /sys/fs/cgroup as cgroup2
  • install Docker and Nvidia runtime as described by @lpdink
  • run a container with nvidia gpu

I also tested Docker Desktop based on their doc: https://www.docker.com/blog/wsl-2-gpu-support-for-docker-desktop-on-nvidia-gpus/ Same results, all good too image

Hope this helps and hopefully more people will be able to confirm it works now too (so we can get back to cgroup2 😇)

This issue occurs after updating wsl from 1.1.6 to 1.2.0.

I am having the same issues, anyone else got this working on WSL 2.x ?

> wsl --version
WSL version: 2.0.14.0
Kernel version: 5.15.133.1-1
> docker run --rm --runtime nvidia --gpus all pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Even though running the nvidia-smi command works fine, outputs all the correct GPU information:

 nvidia-smi
Tue Mar  5 13:33:31 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.10              Driver Version: 551.61         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
...

Hope someone is able to help me out, thanks a lot! Edit: even after downgrading WSL to 1.2.3.0 (even tried to downgrade to 1.1.6.0), the issue persists for me:

> wsl --version
WSL version: 1.2.3.0
Kernel version: 5.15.90.1
> docker run --rm --gpus all ubuntu nvidia-smi

Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Inside the file /etc/nvidia-container-runtime/config.toml change no-cgroups from true to false

wsl --version
WSL version: 2.1.5.0
Kernel version: 5.15.146.1-2

This worked for me, thanks! I actually was even able to remove kernelCommandLine = cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 from .wslconfig

WSL version: 2.2.4.0
Kernel version: 5.15.153.1-2


Thu Aug  8 01:16:02 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.27                 Driver Version: 560.70         CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+

Oh, I have been obsessed by this very bug for couple of days! Thanks you guys for letting me know what the problem is! I have tried wsl 1.2.3 just now and It works very well! 😃

@l1377687647 I noticed that the latest version 1.2.3 has fixed this issue, but it has not been pushed to the Microsoft Store yet, you can download it in release and install it in PowerShell (admin mode) with the Add-AppxPackage command. You may need to call wsl --shutdown at first and terminate the wsl process via Task Manager. Or downgrade to 1.1.6 as above, please note that 1.1.6 is still the version of wsl2 instead of wsl1.

Yeah, very cool! Do you mind telling me how to downgrade to a previous Version of WSL (1.1.6) please? I have not been able to find that out yet.

Hello same problem with 1.2.5.0 can you give some tips to downgrade to 1.2.3 or 1.2.4 ?

@maurange well, just download 1.2.3 or 1.2.4 .msixbundle in release. Open powershell with admain mode:

# find wsl package full name
get-appxpackage *linux*
# uninstall wsl, may need to stop wsl service in task manager.
remove-appxpackage your_full_name
# install 1.2.3/.4 package
add-appxpackage  Microsoft.WSL_1.2.3.0_x64_ARM64.msixbundle
# run wsl again
wsl

if nothing wrong, this will work:)

I am using wsl 1.2.5 and facing this issue. Is wsl 1.2.5 broke GPU driver again?

Same error here, did you fixed your problem after downgrading it?

Its fine. Just figured it out. Just go to the GitRepo and download the msixbundle from the previous version and just double click it. grafik