WSL: vhdx files becoming corrupted since 2.0.4.0 pre-release install
Windows Version
Microsoft Windows [Version 10.0.22621.2361]
WSL Version
2.0.4.0
Are you using WSL 1 or WSL 2?
- WSL 2
- WSL 1
Kernel Version
5.15.123.1-1
Distro Version
Ubuntu 22.04
Other Software
No response
Repro Steps
I can’t really reproduce this, I was mostly wondering if anyone else has had their virtual hd’s gradually becoming corrupted with the pre-release version of WSL2.
I installed v2.0.4.0 two days ago, had an Ubuntu 22.04 (the standard distro from the store) already installed. After some time I suddenly started getting filesystem read-only errors. I’ve had this distro installed (same vhdx file) for almost a year now, never had similar issue (or any other for that matter). Only things that had changed now is the pre-release version and 2 settings activated in .wslconf, autoMemoryReclaim=gradual and sparseVhd=true (I also ran --manage --set-sparse true for my already existing image).
Google said it’s probably a corrupted disk image, e2fsck found errors and supposedly repaired them, but the read-only errors persisted. I loaded a copy of the vhdx file that I had from before the pre-release install, soon enough it also started throwing read-only errors. Debug console showed the root filesystem was being mounted with errors, I would correct them with e2fsck but they persisted.
I ended up nuking both files, and did a fresh distro install. Went fine but at some point it started throwing corruption errors too, this time dpkg wouldn’t run because of corrupted files.
I’ve now spent the last two days uninstalling and reinstalling WSL and testing out distros, but have been having the same issue. I wonder if it could be the sparseVhd option. Has anyone else had any similar issues? I’ve deactivated the option for now and watching if I get a corrupted file again, if I do I’ll probably revert to the previous release version.
Expected Behavior
Mainly I’d expect my vhdx files not becoming corrupted 😅
Actual Behavior
They became corrupted.
Diagnostic Logs
No response
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 17
- Comments: 43 (1 by maintainers)
That’s just outrageous! I was going mad for a whole week with minimal amount of sleep while troubleshooting this super odd issue myself.
Here is my story: At some point in time decided to enable that sparse feature on my WSL2 - it sounded like a super nice feature to have, despite the fact it’s under the experimental section. Some time passed and that home server had to be rebooted. Unfortunately, this was unsuccessful - black screen appeared when the Windows login screen should have. No Windows automatic repair worked, no Windows repair from a media worked, so had no choice but to proceed with a clean installation (fortunately was able to save my files by connecting the corrupted SSD (Samsung 860 EVO 250GB) to another PC). Successfully reinstalled Windows, then started restoring everything, then suddenly out of nowhere file system got corrupted. OK, repaired it, then it got corrupted again. At this point I thought this SSD is gone and installed an old SSD drive (Samsung 840 EVO 120GB) I have. Miraculously, this older drive worked like a charm with that feature for 3-4 days before I decided it’s time to scale the size up and bought a brand new SSD drive (Samsung 870 EVO 500GB). Thought it would be like a walk in the park to clone my old stable drive to the newly bought one - how wrong was I. Cloned disks multiple times with multiple types of software solutions made specifically for that reason with different copying methods - saw plethora of boot BSODs before finally a typical one like 0xc000000e appeared which was actually fixable (https://medium.com/@sergiointoronto/how-to-fix-windows-boot-failure-0xc000000e-c0262352381f Thanks, Sergio!), finally booted successfully and started again to restore things, then BOOM! - sudden file system corruption again - on a brand new drive. Oh, boy, I thought through everything else possible on software and hardware level: *Thought disk cloning is not working good for my case scenario somehow, decided to go with a clean installation again - unsuccessful. *Thought SATA cable is faulty - reseated it, looked in a really good shape anyways. *Thought SATA controller inside the chipset, hence the chipset itself it getting too hot - placed a small passive radiator over it for better heat dissipation - went with another clean installation (because it’s the golden standard according to some very stupid people on MS forums) - again unsuccessful. *Found multiple reddit topics with the same unexplainable behavior - no good comments or no comments at all were found there - no one of the original posters replied how they resolved their situation. *Planned to buy a replacement server - hopefully didn’t order it.
Through that last day of Windows desperate reinstalling was thinking about that sparse feature I’m enabling with my every try. Thought nah, it’s highly unlikely, it affects only the WSL virtual machine. After all my failed tries, decided to search for something like “wsl sparse file system corruption”, and here I am here - pissed as hell.
After a quick clone (quickly stopping the WSL startup automation, disabling this devil feature and recreating my Ubuntu virtual machine) and additional 1-2 hours or restoring, it’s finally good and stable like before. Even survives a reboot, I just cannot believe it!!!
Dear @Microsoft, you’re in debt to me for my 1 week almost 24h/day labor + a brand new SSD! Reach out to me for compensation details.
I’m just really frustrated with Microsoft right now… Who would have thought such available non-hidden (otherwise good) feature for virtual systems would affect the host and the whole system in such unexpected way?! Hope the above helps someone like me.
P.S. Still not sure why it worked well with my old 120GB SSD drive. May be that’s what MS devs are using and this would be the reason to be unable to fix this yet.
Reproduced it again on Win 11 WSL2 setup with host corruption. Currently, sparse vhd is critically dangerous feature!
I’d like to point out that the new Graphical WSL Settings do expose this feature, with no word of warning regarding its experimental nature, let alone the destructive implications.
I don’t know how many people among those that use WSL2 actually use this flag (because it’s an additional step to do, and they must know that this flag exists in the first place), but I guess it’s the minority, which might be part of the reason there’s a small number of issues around it.
Another point is that those that have the host corrupted usually won’t think it’s because of WSL, but due to some hardware or OS issue. I would expect that enabling an experimental flag in WSL would at most (in the most pessimistic case) corrupt the WSL specific files, not the host.
docker probably just creates and deletes A LOT of files, so it gives the sparse stuff a lot more work to do and a lot more chances to make a mess.
It has been almost a year since this feature has been released and for what it looks like we have no fix to the corruption problem yet?
The recommendation at https://github.com/MicrosoftDocs/WSL/issues/1855 should at least be changed.
seriously high paying jobs and still not able to fix such issues in a long duration or probably not working on issue.
Another related issue is that due to hard shutdowns, sudden reboots and other issues WSL2 ext4 filesystem inside *.vhdx could gradually get corrupted over time, so
fsckshould be run first before manualfstrimand manual compaction. I wonder if filesystem corruption was considered while implementing sparseVhd feature in WSL2.This is still an experimental feature for which you need to go out of your way to set a flag. And then when your disk corrupts, you also need to notice it, and make the connection that this is the cause.
For me, on multiple systems, turning on the sparseVhd feature very reliably corrupts the filesystem of both guest and host, so I highly doubt it’s system dependent.
So I’ve managed to somewhat curb the corruption
Noticed that “sparseVhd” put under extra protective flag in WSL2 2.5.6 version. Looks like the issue was accepted as existing: https://github.com/microsoft/WSL/releases/tag/2.5.6
I’m working on finding a reliable way to recover lost data as a result of this. If you’ve lost data you want to recover please don’t use the drive.
Also, did anyone else find reliable ways to recover the data stored on their VHDX?
Yes, I’ve noticed this issue while playing with docker and fstrim commands.
Just FYI.
I was also facing this issue and gave up using sparseVhd. To me, it seemed using docker made corruption frequently. Docker may not be the direct cause, but it can be a key to reproduce this issue.
My VHDXs are showing 0 bytes also, can the data be recovered?
I enabled sparseVhd a few months ago, for the last month my filesystem in ubuntu started getting corrupted every other programming session, especially when running docker desktop. Prior to enabling this option, I had been using this system for over a year and had no problems. Also, the windows file system started to get corrupted when using wsl too, in cases where I don’t use wsl, this behaviour is not observed.