WSL: WSL complete freeze

Version

Microsoft Windows [Version 10.0.22000.918]

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

5.15.57.1

Distro Version

Ubuntu 22.04

Other Software

PhpStorm 2022.2.1

Repro Steps

  1. Open a large project in PhpStorm (starts indexing project fileds…)
  2. Open another large project in PhpStorm (starts indexing project fileds…)
  3. Keep switching between large projects and sooner or later WSL is completely frozen while PhpStorm is indexing files

Expected Behavior

PhpStorm finishes files indexing and WSL is usable.

Actual Behavior

WSL freezes completely including any wsl.exe command, so even wsl.exe --shutdown will hang forever. The only way to restart WSL is by doing a computer restart.

Diagnostic Logs

Uploading via Feedback Hub.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 23
  • Comments: 70 (1 by maintainers)

Most upvoted comments

I am the original reporter of this issue and it has been fixed for a while already. WSL is not dead at all, am using it daily at my day job. Your problem is a different problem, please open a new issue.

Thanks a lot @michaelarnauts and @schiorean. With the debugging information you shared we have identified the root cause.

The issue is caused by bug in MUSL which causes a deadlock in init.

This bug only occurs if fork() and aio_* methods are called at the same time AND if malloc() needs to increase the process’ memory, so it’s pretty rare.

The issue has been reported to the MUSL maintainers, but a fix hasn’t been released yet. In the meantime we’re working on a workaround inside WSL to avoid the potential deadlock.

I’ll update this issue once the workaround will be released.

I have the same with WSL version: 1.0.3.0 Kernel version: 5.15.79.1 WSLg version: 1.0.47 MSRDC version: 1.2.3575 Direct3D version: 1.606.4 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.19045.2364

Happens mostly in Pychar,/Intellij. Sometimes in terminal

Considering to stop using wsl in favour of Linux itsef. If it haven’t fixed after an year seems Microsoft spoiled technology and doesn’t know what to do.

I’m still getting this issue happen to me - I’ve wiped and reinstalled Windows 3 times, had Dell replace hardware like the battery and hard drive. Issue only happens after I’ve installed WSL, and then it occurs multiple times a day.

Hi @OneBlue,

I reproduced it again, but I don’t have the gcore output because I can’t figure out how to install gdb in the system shell (none of the usual commands are recognized e.g. trying apt install gdb I get -bash: apt: command not found). So until I can install gdb in the system shell (I sent an email to [edited] I am sharing here the output of dmesg and wslservice service dump.

Some extra notes, maybe it’s helping:

  1. Even though wsl.exe is frozen the currently open shells are still usable (btw, can’t I install gdb in the normal, non-system, shell and provide the dump from there?)
  2. The only way to restart WSL besides a full computer restart, is by opening a terminal as Administrator, then run hcsdiag list followed by hcsdiag kill.

I tried to upload the zip file containing the dumps here, but looks like File size is too big. I uploaded again in Feedback Hub https://aka.ms/AAi265d however when I open Details I can’t see it… If you can’t see it in Feedback Hub please tell me where to send the file?

Happens for me, too. WSL freezes completly while consuming 100% CPU.

Same problem, same versions for everything. It looks like our problem is being tracked over here

I uninstalled any WSL Windows updates, removed my distro, disabled Virtual Machine Platform and WSL features and then installed everything again including the latest WSL from the store. I also gave WSL 6GB of RAM instead of 4GB that I was using in .wslconfig. Now it seems I’m not hitting the issue anymore, but unsure which thing exactly has fixed it.

The thing is that the issue just started to appear for me a few weeks ago. Before that, everything worked fine.

Now, using Docker + VS Code, breaks quite frequently during the day. The only workaround for me is stopping and starting wslservice, but it breaks my workflow, totally.

I will try reinstalling Ubuntu, but there are other people having this issue, see also #9114.

Why was this issue closed I’m still having this issue? Please reopen it.

gcore -a $(pgrep init) [New LWP 5] 0x00000000003bc9cd in ?? () Saved corefile core.1 [Inferior 1 (process 1) detached] [New LWP 28] warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable. Connect to gdbserver inside the container. 0x00000000003bc9cd in ?? () Saved corefile core.2 [Inferior 1 (process 2) detached] 0x00000000003bc9cd in ?? () Saved corefile core.6 [Inferior 1 (process 6) detached] [New LWP 27] warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable. Connect to gdbserver inside the container. 0x00000000003bc9cd in ?? () warning: target file /proc/26/cmdline contained unexpected null characters Saved corefile core.26 [Inferior 1 (process 26) detached] 0x00000000003bc9cd in ?? () Saved corefile core.31 [Inferior 1 (process 31) detached] 0x00000000003bc9cd in ?? () Saved corefile core.32 [Inferior 1 (process 32) detached] root@MarkSpectre14 [ /mnt/c/Users/mlwp/Software/WSL/10_29_2022 ]# ls core.1 core.2 core.26 core.31 core.32 core.6

wsl_hang.zip

@OneBlue

> wsl --version
WSL version: 0.68.4.0
Kernel version: 5.15.68.1
WSLg version: 1.0.44
MSRDC version: 1.2.3401
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.674

core.zip (all coredumps) wsl dump.txt (all terminal output)

If you’re not enrolled in Windows insider, that makes sense since we haven’t published that package everywhere yet.

You can install it without an insider account by downloading the package and running something like (elevated PowerShell) :

$installedPackage = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $installedPackage -AllUsers
Add-AppxPackage /path/to/msixbundle

try

wsl --update --pre-release

wsl.exe --shutdown works for me but I am unable to run anything for a long time on this. I wonder if people experiencing this are utilizing multiple cores? Seems to happen to me when I run a CPU intensive program (CPU usage going to 100%)

Since I upgraded to 0.70.4 I didn’t experience any freeze. All good for me.

Thank you @schiorean.

The fact that you managed to reproduce the issue with a different kernel shows that it’s not a recent kernel upgrade issue at least.

To give a bit of context what seems to be happened is that the init process is stuck during a syscall while creating a new session. To understand why, we’ll need to look at the kernel stacks.

So do this, can you please:

  • First clear everything by running wsl --shutdown (make sure nothing is running). If this is unresponsive, kill wslservice.exe.

  • Then (if not already done) upgrade to wsl 0.68.4 by running: wsl --update and undo the changes to .wslconfig (meaning revert to the official kernel)

  • Open a shell inside your the system distro (and leave it open) via: wsl -u root --system

  • Install gdb in that shell via: tdnf install gdb (You’ll need to do it again because the system distro is not saved between reboots)

  • Run the steps you described to get WSL into that “frozen” state in your regular distro (not inside the system distro)

  • Once WSL is frozen, use the previously opened shell to dump all the ‘init’ processes via: gcore -a $(pgrep init) This will generate a few core.* files, please shares those on this issue

  • Then please share the output of (Still in the system shell) for pid in $(pgrep init) ; do echo -e "\nProcess: $pid" && for tid in $(ls "/proc/$pid/task") ; do echo "tid: $tid" && cat "/proc/$pid/task/$tid/stack" ; done ; done

  • Also please share the output of dmesg and cat /proc/meminfo in the system shell

With this we should have the kernel stacks of all the threads in the init processes

@OneBlue This still crashes daily here. Anything else I can do to help troubleshoot this?

I’m on 0.68.2.0 now, the latest release on Github.

wsl --version
WSL version: 0.68.2.0
Kernel version: 5.15.62.1
WSLg version: 1.0.44
MSRDC version: 1.2.3401
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22000.1042

I have the same problem as OP. Same usecase (Phpstorm that causes WSL to hang during indexing, or probably, anything that does high IO). It happens multiple times a day, and I can reproduce this quite easily.

A faster way to get a fresh WSL env without rebooting (since the wsl --shutdown indeed hangs forever) is to kill wslservice.exe from the task manager.

The output of collect-wsl-logs.ps1 is here: WslLogs-2022-09-28_14-05-40.zip

wsl --version:

WSL version: 0.66.2.0
Kernel version: 5.15.57.1
WSLg version: 1.0.42
MSRDC version: 1.2.3401
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22000.1042

I’ll try the “unpublished” package above and retry.

Thank you @schiorean.

With this information, our current theory is that this issue was introduced by the latest Linux kernel upgrade.

To validate this, can you please:

[wsl2]
kernel=C:\\path\\to\\kernel-5-10

(Make sure that the ‘\’ are doubled)

  • Then shutdown wsl via: wsl.exe --shutdown
  • Run: wsl uname -a to make sure that the correct kernel is in use. It should say: Linux [hostname] 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  • Then try to reproduce the issue (please share dmesg output if you can reproduce again with this kernel)

@OneBlue attached are the logs are per your latest instructions (took a 2-3 hours until hang happened). wslservice.exe dump uploaded separately here https://drive.google.com/file/d/19nPGc6-NOpv_f0RSp8-5kgRwW1Y-1y5n/view?usp=sharing

WslLogs-2022-09-26_11-09-22.zip dmesg.txt proc.txt vsoc.txt

Thanks @schiorean.

After looking at all the dumps, I have a good idea of where the issue is, but I’ll need a bit more info to root cause it.

I built a private version of the package with extra logging: https://1drv.ms/u/s!AiWXuqXSX5K2d45U1MHyOchps0k?e=w8SSjC To install it (elevated powershell):

# Remove the installed package
$installedPackage = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $installedPackage -AllUsers

# Trust the private package's certificate (since it's not an official build, it's not signed with the official Microsoft certificate)
(Get-AuthenticodeSignature "/path/to/msixbundle").SignerCertificate | Export-Certificate -FilePath private-wsl.cert
Import-Certificate -FilePath .\private-wsl.cert -CertStoreLocation Cert:\LocalMachine\Root

# Install the package
Add-AppxPackage /path/to/msixbundle

Once the package is installed:

  • before doing anything please start the log collection with collect-wsl-logs.ps1
  • Open a shell inside your the system distro (and leave it open) via: wsl --u root --system
  • Reproduce the hang
  • Inside the system distro shell, run: ss -lap --vsock, ls -la /proc/*/fd and dmesg and share the output of the three commands
  • Dump wslservice.exe

This should give us more information to identify what’s happening.

Actually here’s the wslservice.exe dump as well https://drive.google.com/file/d/19nPGc6-NOpv_f0RSp8-5kgRwW1Y-1y5n

@OneBlue so it happened again with 0.67.6.0. I attached linux core dump and dmesg. wslservice.exe dump file I uploaded via Feedback hub again, if for some reason you can’t access it I will upload it in Google Drive, let me know please. core_take_3.zip dmesg.log

@OneBlue here’s another core dump, this time it happened faster compared with the previous one. I estimate about ~20 minutes since started WSL & system console.

core-take-2.zip