WSL: WSL2 reliably crashes when handling files > 10GB

Environment

Windows build number: Microsoft Windows [Version 10.0.19041.329]
Your Distribution version: Ubuntu Release: 20.04
Whether the issue is on WSL 2 and/or WSL 1: WSL 2
Linux version 4.19.104-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Wed Feb 19 06:37:35 UTC 2020

Steps to reproduce

dd if=/dev/zero of=./dump.bin bs=4k iflag=fullblock,count_bytes count=20G

This will crash at around ~10GB, here are 3 runs:

-rw-r--r--  1 root root 12729712640 Jun 15 16:10 dump1.bin
-rw-r--r--  1 root root  9093251072 Jun 15 16:11 dump2.bin
-rw-r--r--  1 root root 11588861952 Jun 15 16:12 dump3.bin

Host OS has plenty of free space/32GB ram.

Expected behavior

It should not crash the entire WSL host along with possible docker containers.

Actual behavior

Ubuntu/WSL window closes, all processes / docker containers crash with it.

I’m actually working with larger gz files, unzipping them, copying them, etc. Basically every operation touching files > 10GB does not work.

Files mounted via /mnt/c/ do not appear to have this problem.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 19 (2 by maintainers)

Most upvoted comments

While attempting to compile Chromium on WSL2, I encountered a consistent crash issue. After extensive research and troubleshooting, detailed in my blog post, I discovered a workable solution:

  • Allocate as much RAM as possible to your VM and ensure there’s a substantial amount of SWAP space to prevent OOM (Out of Memory) errors.
  • Address the WSL2 crash issue by setting up large pagefiles on Windows—I used 256GB, but 64GB might suffice for compiling Chromium.
  • Avoid utilizing all CPU threads during compilation to prevent WSL from crashing; using half or up to two-thirds of the threads seems stable.

It appears I never paid attention to memory usage.

After limiting memory via .wslconfig to 12GB I can create the 20GB dummy file. https://github.com/microsoft/WSL/issues/4166

Seems like file operations inside the WSL2 filesystem consume/block memory to a point where it crashes. This also explains why the windows mounted filesystem /mnt/c was not affected by it.