WSL: flood of hv_storvsc errors, high cpu usage even if idle
Version
Microsoft Windows [Version 10.0.19044.2311]
WSL Version
- WSL 2
- WSL 1
Kernel Version
5.15.74.2-microsoft-standard-WSL2
Distro Version
Ubuntu 22.04
Edit: and Debian Bullseye, Fedora Remix for WSL 37, with both systemd enabled and disabled.
Other Software
Edit: McAfee Antivirus, Sophos Safeguard. The issue disappeared after I switched antivirus (from McAfee to MS Defender) and management software (removed Sophos Safeguard, but BitLocker still enabled).
Repro Steps
Open Ubuntu (or any other distro – I have seen this in Debian and the Fedora Remix for WSL).
Expected Behavior
Normal performance.
Actual Behavior
High CPU usage from rsyslog and journald, around 15-20% CPU, reflected by high CPU usage by Vmmem. Disabling kernel logging in rsyslog and journald quiets the machine, but I still see some 10% CPU from Vmmem. The cause seems related to the following message flooding the kernel log:
[ 271.065940] hv_storvsc fd1d2cbd-ce7c-535c-966b-eb5f811c95f0: tag#599 cmd 0x28 status: scsi 0x2 srb 0x4 hv 0xc0000001
[ 271.066203] hv_storvsc fd1d2cbd-ce7c-535c-966b-eb5f811c95f0: tag#599 cmd 0x28 status: scsi 0x2 srb 0x4 hv 0xc0000001
[ 271.066412] hv_storvsc fd1d2cbd-ce7c-535c-966b-eb5f811c95f0: tag#599 cmd 0x28 status: scsi 0x2 srb 0x4 hv 0xc0000001
I couldn’t find any info about this particular error, so I am reporting this here.
The only way to stop this, at least that I could find, is to shutdown the virtual machine altogether.
Edit: the above messages are SCSI read errors (see kernel line generating the message).
Diagnostic Logs
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 7
- Comments: 34
Commits related to this issue
- scsi: storvsc: Ratelimit logs If there is a persistent failure, it's possible to flood the logs with errors and warnings, maxing out CPU utilization and preventing attempts at fixes. See [1] for an e... — committed to easwarh/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs If there is a persistent failure, it's possible to flood the logs with errors and warnings, maxing out CPU utilization and preventing attempts at fixes. Closes:... — committed to easwarh/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service If there's a persistent error in the hypervisor, the scsi warning for failed IO can flood the kernel log and max out CPU utilizat... — committed to intel-lab-lkp/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service If there's a persistent error in the hypervisor, the SCSI warning for failed I/O can flood the kernel log and max out CPU utiliza... — committed to intel-lab-lkp/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to mj22226/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to mj22226/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to mj22226/linux by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to linux-ia64/linux-stable-rc by easwarh 2 years ago
- scsi: storvsc: Ratelimit warning logs to prevent VM denial of service commit d2138eab8cde61e0e6f62d0713e45202e8457d6d upstream. If there's a persistent error in the hypervisor, the SCSI warning for ... — committed to Whissi/linux-stable by easwarh 2 years ago
The cause was a bad block on the Windows host SSD. When WSL tried to read the part of the VHDX corresponding to the bad block, it appeared to get stuck in this condition.
Same here, getting more than 2000 error messages per second, battery is draining fast.
Edit:
Does anybody know if this problem occured in earlier versions of WSL? If not, is there any way to downgrade?
Because it would explain why I face problems with WSL, but some of my colleagues don’t. I didn’t make that connection until you mentioned it, but now it seems obvious.
Would this be lancrypt?
So far I’ve found the following kernel versions work for me (i.e. stop the flood of hv_storvsc messeages).
I’m currently trying to locate a build of 5.10 kernel to test as well (I tried building it from source but it failed for some reason)Edit: I was being blind yesterday, and found the 5.10 and 5.15 releases on the carlfriedrich repo https://github.com/carlfriedrich/wsl-kernel-build/releases
Edit2: see my comment below for the cause of the issue on our setups https://github.com/microsoft/WSL/issues/9173#issuecomment-2808686068
My workaround was to install a new WSL distribution, then extracting files from the broken distribution by adding
--mount C:\Path\To\Broken\ext4.vhdx --vhd --barewhen invokingwsl.exe, and then in the new distribution runningsudo mount -t ext4 -r /dev/sdX /mnt/broken_vhdx/so that I could start copying fileshttps://github.com/carlfriedrich/wsl-kernel-build/issues/1 - try older kernels.
I don’t think this is always a hardware issue. I tested all drives involved where I was seeing this and found no errors. Currently I’ve scheduled the linux VM to reboot every day and Windows once a week. I also saw a hyper-v patch in one batch of updates but didn’t dig further. Not sure if either of those have “fixed” it, but I haven’t seen this error in over a month now.
My wsl crashed the other day,I can’t login to any distribution, use
wsl -t archto terminate andwsl --unregister archto unregister archlinux, it completely stuck without any response, after that I have to restart my computer, and today I find 2 very big log files in my wsl ubuntu , :I found that the last hundred lines of the two log files kept repeating the following:
Feb 5 00:00:16 ... kernel: [141600.214394] hv_storvsc fd1d2cbd-ce7c-535c-966b-eb5f811c95f0: tag#511 cmd 0x28 status: scsi 0x2 srb 0x4 hv 0xc0000001I’m not sure, but I believe they are related, I enabled systemd, I’m not sure if antivirus has anything to do with this, I think I didn’t run a specific programm that day, it just appeared suddenly, and crashed