WSL: WSL2 intermittently slows/hangs on Windows on ARM

Windows Version

10.0.22621.2428

WSL Version

1.2.5.0

Are you using WSL 1 or WSL 2?

  • WSL 2
  • WSL 1

Kernel Version

5.15.90.1

Distro Version

Ubuntu 22.04

Other Software

  • Docker Engine (installed in WSL following the instructions for installing on Ubuntu). Not Docker Desktop.
  • VS Code with the Dev Containers extensions, for enabling development within a Docker container.
  • .NET 7 SDK, installed in the development Docker container started by VS Code’s Dev Containers extension.
  • MongoDB running in Docker container in Ubuntu on WSL2.
  • OpenTelemetryCollector running in Docker container in Ubuntu on WSL2.

Repro Steps

Start a VS Code remote development environment with a development Docker container running on Docker Engine on Ubuntu on WSL2. Start working/compiling/debugging and observe that the WSL2 environment, including all Docker containers running on WSL2, will start to slow down/hang with high CPU usage after a short while (specifically the VmmemWSL process). The environment will completely hang for a number of seconds, then continue working for a while.

The problems not only affect the Docker environment but also “native WSL2” shells/environments.

The problems get increasingly worse (more frequent hangs which lasts for longer and longer time) until the WSL2 is completely unusable. Most of the time, running wsl --shutdown will shut WSL2 down, after which it can be restarted, and the slow/hang cycle can begin again. Note that running wsl --shutdown takes abnormally long to execute when this issue has been encountered (30s to 10 min).

More complex development environments (for example multiple code projects compiling in parallell, development databases and other software running in containers) will speed up the slow/hang cycle.

This issue is created per request in issue #9454.

Expected Behavior

The VS Code remote development environment will not slow down/start to hang over time and will continue using “normal” CPU cycles.

Actual Behavior

The VS Code remove development environment, and WSL2 as a whole, starts to slow down/hang with increasing frequency.

Diagnostic Logs

WSL dumps collected during a hang is available at https://doits-my.sharepoint.com/:u:/g/personal/david_nordvall_doitsolutions_se/EedqfOgnSnJOtLAKe7Jd2R0BqtdtieIthWXUbvLOPBf9zg?e=2N8fuq.

Please note that I got several instances of this error when the dumps were collected:

Line |
 118 |          $Result = $DumpMethod.Invoke($null, @($process.Handle,
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | You cannot call a method on a null-valued expression.
Failed to write dump for: C:\Users\DavidNordvall\Downloads\wsl-dumps\WslLogs-2023-10-23_19-15-49\dumps/wsl.7204.dmp
Writing C:\Users\DavidNordvall\Downloads\wsl-dumps\WslLogs-2023-10-23_19-15-49\dumps/wsl.12444.dmp

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 30 (2 by maintainers)

Most upvoted comments

@david-nordvall - yes it is a hypervisor fix specific to running virtual machines.

Since that time I have had wsl hangs every 20 minutes, as it was before. So switching of graphics support eliminates some warnings in kernel logs, but not making overall situation any better. Being a kernel engineer in a mean time, I decided to add some additional debugging features to WSL2 kernel so this lockups could be somehow investigated. For that I have to rebuild the kernel from the sources, altering it’s configuration. I tried to clone kernel sources but wasn’t able to finish this process any time I tried because WSL VM hangs every time. Luckily I have another arm machine, so then I decide to buld kernel there and just copy the kernel to my arm windows notebook. Also I found a recipe how to build latest ( mainline) kernel for WSL – https://github.com/oleksis/wsl2-linux-kernel-config So I built kernel verison 6.8-rc4, copied this kernel to my notebook and after modifying .wslconfig and rebooting WSL VM, I was able to clone and build kernel sources in WSL VM, that I can’t do before. There were some RCU panics in kernel logs, but they don’t lead to complete WSL lockup, system recovered by itself. I will continue to invesitagate those panics, but if you’re brave enough you may try to run your WSL2 VM with 6.8 kernel – I uploaded it to my Dropbox. Here is the link – https://www.dropbox.com/t/MpRiV3DJsrtKE1U3 Here is what .wslconfig should look like:

[wsl2]

kernel=d:\\Image

gpuSupport=false

debugConsole=true

@vadika did you happen to book any progress on this, I personally haven’t had the time to dig into it further.

No, unfortunately – the problem is somewhere out of my scope of understanding, available code is not enough to debug it. I have made several attempts to localize the problem, and seems it’s somewhere between VM kernel-hypervisor interaction, probably some communication channel is dropped from HV side makes kernel to deadloop.

There is a Windows OS fix that targets an ARM-specific hang that should be going out next month (I think the fourth patch Tuesday of November).

@vadika Yes, I’ve been testing it over the weekend and unfortunately the issue most definitely persists. 😦

@vadika, I did give your kernel a quick try but it resulted in numerous crashes of the type I haven’t seen before. In other words, it may well fix the hanging issues but, for other reasons, it didn’t work long enough for me to verify 😃.

It sounds great that you will continue to investigate the panics!

So, I realize Windows on ARM isn’t Microsoft’s biggest priority. Especially issues specific for developers using Windows on ARM, since there probably are very few such developers. But I’d still love to hear if the Hyper-V fix you mentioned, @benhillis, has been released yet?

Considering that there “soon” will be devices with Qualcomm’s new Snapdragon X processor, which might appeal to more developers, and that Docker might release a Docker Desktop version for Windows on ARM during 2024, I’d hope Microsoft would pay a little more attention to this issue, since it pretty much makes Windows on ARM a non-starter as a developer platform, regardless of the processor’s underlying performance and efficiency.

Another patch Tuesday-patch applied. The issue still persists:

WSL-version: 2.0.9.0
Kernelversion: 5.15.133.1-1
WSLg-version: 1.0.59
MSRDC-version: 1.2.4677
Direct3D-version: 1.611.1-81528511
DXCore-version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows-version: 10.0.22631.3085

@benhillis One more patch tuesday-patch applied and the issue still persists. Any word on if the hypervisor fix has actually been released? If it has, how can I help you get more data for debugging? This issue really needs to get fixed… 😦

Current WSL version info:

WSL-version: 2.0.9.0
Kernelversion: 5.15.133.1-1
WSLg-version: 1.0.59
MSRDC-version: 1.2.4677
Direct3D-version: 1.611.1-81528511
DXCore-version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows-version: 10.0.22631.2861