WSL: Windows Defender/Anti-malware Causing Performance Issues after CU update

So, after uninstalling my old Ubuntu Bash lxrun /uninstall /full and after the update lxrun /install I noticed, as promised, that processes running in WSL could be seen in the Windows task manager (yay!); however, there is an issue that consistently causes Windows Defender/Anti-Malware service to consume over 50% of CPU utilization whenever code compilation or builds are happening inside bash. Back last version (before the processes were visible to Windows – or more sandboxed, not sure) this was not an issue at all. So I’m wondering if this is a worthy trade off? I, for one, as a dev, would prefer to maximize build/compile performance even if that means reverting to a more sandboxed environ (or at least the option to do so).

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Reactions: 78
  • Comments: 131 (13 by maintainers)

Most upvoted comments

Adding an exclusion does NOT fix the performance problem. Adding an exclusion for VS does NOT fix the performance problem. Adding an exclusion for IIS does NOT fix the performance problem.

The only solution that works for us is to disable REALTIME protection in Windows Defender.

I also want to note that the performance impact is DRAMATIC.

For instance, a rebuild of a VS2017 solution that takes ~20seconds to complete with REALTIME protection turned off, takes more than 3 minutes to complete with Windows Defender REALTIME protection enabled.

Another example: a GIT local check-in with one file changed, completes almost instantly with Windows Defender REALTIME protection DISABLED. This same checkin takes more than 2 minutes with Windows Defender REALTIME protextion enabled.

The exclusions DO NOTHING to improve the situation.

Please bear with us. We are investigating. If it was a simple issue to fix, it’d have been fixed by now 😉

For what it’s worth, I always add an exclusion for my big source-code build directories to Windows Defender, for exactly this reason. I trust my own code, and the performance hit is too big otherwise.

@bluddy We hear you and share your pain. @tara-raj & team are looking into this issue, but any fix in this area will require great care, so will take a little while. But we ARE keen to improve this scenario.

Thanks for your patience.

Here’s another workaround which works for me: for each directory in your linux path add a process exclusion rule with the equivalent windows path with a wildcard. So, instead of excluding just bash with a process exclusion rule of C:\Users\ipc\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\bin\bash use C:\Users\ipc\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\bin\*. I did that for /bin, /usr/bin, /sbin, /usr/sbin… basically every linux dir in my $PATH that had an executable in it. I also added a folder exclusion rule for the base dir of the installation because that prevents the executable itself from being scanned by real-time (see this page for details).

With that change I can leave real-time scanning on; my compile times in WSL are just as fast as they are when Defender’s real-time scanning is disabled and Defender no longer spikes the CPU as much. Defender still takes up some CPU time but it didn’t seem to affect my build times so I’m not too concerned.

Here’s a powershell script that I was using to add my rules. Change the username and package name to your own usernames and package.

Does this work for anyone else’s test case?

The status is as it was when I last replied to this thread:

I appreciate this is frustrating, but the fact that it’s not yet resolved is just a glimpse into how complex this issue is.

For those who’re interested, #873 is the main issue tracking disk perf issues in general, and you’ll find more commentary, and detailed feedback there, including from the engineer responsible for most of WSL’s disk IO engine.

You can be sure that as soon as we have a decent solution ready for testing that this issue will be updated, and there will be much rejoicing on Twitter and elsewhere.

Until then, please bear with us while @tara-raj works across several teams to figure out and implement a decent solution.

Thanks.

#somethingwehavetolivewith #itisnotthatbad #iuseitdailyanditsfine #justsayin

it’s been solved in WSL2 go join insiders and enjoy the power…

Same. None of the suggestions work for me, too - on my primary PC, which I don’t have admin rights to (therefore cannot use these workarounds), these workarounds are hopeless. On my secondary PC, I’ve tried it, and got similar results to @colinmollenhour . If virtualbox can do it, then it’s certainly possible (yes, virtualbox requires a driver - but the performance should still be in the same ballpark. Please don’t do anything to WSL that makes it require admin rights on startup/for simple tasks, the guy who as admin rights to my computer is very rarely around).

hoW aBoUt opTimIze WiNdoWs defeNDEr, oR mAkE It noT SCAn EveRy sIngLe file BY DEFAULT (IT hAS tO NOT DO ThiS BY DEFAULT becauSe THeRE ARE sTANdArD uSERS OuT tHEre, AND somE, LiKE mE, WiLl gO fOr 6-7 MoNtHS At a TiMe wIThoUt COntAct WiTH tHe PerSon wHO hAs tHE aDmIn accOuNt)? Windows defender’s probably pretty optimized already tho.

Thanks @ian-p-cooke I think I can finally use git efficiently in WSL. Thanks!

@spences10 The executables can vary a bit, but 99% of the time they’re in standard locations. Starting from the WSL root filesystem directory in Windows, which is something like C:\Users\YOURUSERNAME\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs the paths to the Linux executables will be be at:

  • bin\*
  • sbin\*
  • usr\bin\*
  • usr\sbin\*
  • usr\local\bin\*
  • usr\local\sbin\*

This is using Ubuntu - the list might vary in CentOS or other distro.

If you have special cases, you could add those too. For example Ian Cooke’s powershell script also excludes home\$linuxuser\.cargo\* - I don’t know what .cargo is for, presumably some build tool that he uses. You probably don’t have that and can omit it.

At this point I have the windows defender config panel open in the background, and I just repeatedly go to it and turn real time protection off while I’m working. It’s a real pain, and it needs to be fixed. Not to mention that it leaves my PC vulnerable.

@firewave Strongly recommend dumping McAfee if there is any possible way. Windows already has a built-in anti virus and, to put it lightly, its more trustworthy.

try the major brands like trend micro, sophos, mcaffee, crowdstrike. i won’t recommend one as there would be the danger of MS buying and destroying it, as they do with everything they buy.

The team is looking into this and will continue investigating. FWIW this rolls up into a broader filesystem perf improvement endeavor.

In the meantime if you can provide us with more specifics so we can generate a repro that would be very helpful. @ow which repo are you using? @justinmchase which errors do you hit?

Any update? Windows defender still slows apps first time ran on win10 and win11.

@lewisdonofrio Yeah, that’s pretty unfair—I can’t work on a number of my repos in the current scenario without disabling Windows Defender entirely, which I’m doing daily, because they take 5-10 minutes to build vs less than 30 seconds without. This is a real problem, and discounting other people’s well-documented problems with it isn’t helpful.

as @bitcrazed stated “Dear WSL fans, despite the perf impact of Defender (which we are working to address), THIS is why we DO NOT recommend excluding folders from or disabling Defender” defender-wsl-good (https://twitter.com/richturn_ms/status/1095207032292929536)

As an FYI. We use the WSL Switcher to use centos under WSL. Adding exclusions helped improve git performance by 50%.

$lxss = $env:LOCALAPPDATA
Add-MpPreference -ExclusionPath "${lxss}\lxss"

For those here having the same issue, I’ve just completely given up on Defender pinning my CPU all the time. Would prefer to have it on, but currently nigh unusable on my setup.

If you want to force Realtime protection off, you can use this registry key, since Microsoft insists that it will turn itself back on within a few hours on Fall Creator’s and up.

Paste below into a file, add .reg extension, double click. Reboot and boom, hey, it’s actually usable.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender]
"DisableAntiSpyware"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows Defender\Real-Time Protection]
"DisableBehaviorMonitoring"=dword:00000001
"DisableOnAccessProtection"=dword:00000001
"DisableScanOnRealtimeEnable"=dword:00000001

for the love of god, switch to a different AV scanner. defender is a steaming pile of dog turd in terms of performance and efficiency.

i consider this as constructive feedback and a positive contribution to this thread as my proposed solution will solve the mentioned issues.

The Defender team are currently assessing how/what/when/if Defender on the host interacts with workloads running in VMs and/or Defender running IN VMs. I do not yet have anything to share on what the final outcomes will be, but know that we’re working closely with them on this matter.

Just want to throw in there something positive for a change—after all of this thread and the frustrations with slowness, Microsoft really came up with a great solution in WSL2, and I’m pretty excited about what that means going forward. I’ll have to break my own no-beta rule and install Insiders for this, because it’s going to change everything!

A few people in here have said that adding an exclusion in Windows Defender does nothing but I wanted to go ahead and give it a try anyway. A quick google search for a directory that I should exclude was C:\Users\{user}\AppData\Local\lxss, but pretty sure that was legacy.

Most instructions I could find, granted was a quick search all pointed to excluding a directory, but I noticed that Windows Defender Security Center also allows excluding process. So I excluded

Apt
Bash
Dash
Dpkg
Init
Mandb
Sudo
Ubuntu

And noticed an immediate improvement. Although it is not blazing fast, it has sped up a little bit during this huge install it was already working on. Also for the first time during this install, I am not seeing my disk constantly at 100%. I’m at 58% right now, yes - I have seen it go back up into the high 90s but it goes back down. This is much preferred to constant use.

As for the speed increase. I cant say for sure. I was mid install but it definitely seems like its going faster. Even if not, I will take the disk saving. One thing I also noticed is there is a subprocess of Ubuntu, ubuntu - but Windows Defender will not allow adding Ubuntu and ubuntu. Not sure if that’s an issue.

@tara-raj I can get the poor performance to occur on almost any repo.

Do a git clone on something like this - the performance will be slow in general, but can push it further by committing new changes or just installing with npm - in all seriousness I’m turning Defender off 6-8 times a day to get the performance bottleneck out of the way. I’ve not timed it, but I’d hazard a guess it’s more than 4x as bad with Defender enabled.

There’s a ton of documentation about this across repositories — Yarn has a bunch of ongoing threads about this and has been writing workarounds to handle it. Generally speaking, WSL works great, but I will say these performance issues have been crippling and frustrating.

In my experience, adding the WSL Linux filesystem root to the directory exceptions for Defender is ineffective, or at least, doesn’t make a significant difference to the AV chewing the CPU inspecting running processes. The only method I’ve found effective is adding each WSL Linux process I use commonly to the process exclusions.

Just to reinforce this (old) issue that isn’t solved yet, I just did a sudo apt upgrade. Look at this picture and see who is always fighting the Dpkg process for resources:

image

If I’m compiling something, it is even worse… I think the exclusion of the source directory is being totally ignored.

I understand that disable Windows Defender on registry is not the proper fix as we would have other security issues later on, but have this issue that long open is frustrating. We see more and more fixes/enhancements getting added to WSL but this, which is critical to any use case, is not sorted yet. 😦

The sad part is that this issue was reported over 8 months ago and we’ve yet to see any progress or patch to remedy the issue. Instead of having to disable Windows Defender, they should revert to the way WSL worked before and let it be its own isolated subsystem. Integration with Windows is what caused all these perf problems.

In WSL2, each distro’s filesystem is stored in individual EXT formatted VHDX files which are mounted within the distro’s container. As such, the distros’ filesystems aren’t stored in NTFS like in WSL1.

It’s nothing to do with cheating. It’s all about compatibility and performance.

As I said, we’re working with Defender and several other teams on a number of initiatives to further improve WSL2’s integration, performance, interop, and manageability. Stay tuned to our blog for updates as we make progress.

@ow @anstosa I would say it is safe to assume at this point that there is no easy (or even remotely feasible) fix for this issue in WSL1 and this is one of the top motivations for WSL2 so this simply won’t get fixed in WSL1. No amount of complaining and begging is going to change this.

The terrible performance in WSL also makes it unusable for my work, which is why I’ve chosen instead to… not use it and do my work inside a VM until a better solution becomes available. If WSL is not the best solution then quit using it.

@lewisdonofrio a) it’s not even in insiders for another month or so as far as I can tell and b) of course it’s great that it’s been solved and that some are willing to help test. But it’s super counterproductive to claim that there is an immediate solution right now when taking it involves trading the stability of your entire operating system by joining the fastest insiders track…

I don’t see CPU spinning a lot by the Defender process. I see I/O reads/writes overdose though

Yeah, it feels like adding exclusions is just a placebo - the performance is still incredibly slow for a lot of stuff related to development. But how do we even raise an issue with the windows defender team about this?

This problem is still happening. Iv even noticed, on either, wsl2 or native windows, when i run a cpu intense script (multiprocessing, walking, threading, data-anything etc… windows malware service executable shoots up to 89% and wont let go until i kill the process, immediately goes away. directory exclusions included. I wonder if adding program files/your_lang to exclusions list would work for interpreted languages?

This issue has been automatically closed since it has not had any activity for the past year. If you’re still experiencing this issue please re-file this as a new issue or feature request.

Thank you!

I totally hear you re. management of WSL instances. The fact that WSL1 ran atop NT and thus could be managed and monitored by existing Windows tooling and infrastructure is something many enterprises in particular have expressed considerable appreciation for.

But the reality is that for those using WSL, WSL1’s compatibility and file IO performance just weren’t where they needed to be, so we had to change tack and utilize our new lightweight container infrastructure to deliver high performance and 100% compat.

This does mean, however, that many of the management tools & tech that you’d use to manage, secure, configure, and operate physical Linux boxes and/or VM’s are also likely to work just the same in WSL2!

And as I said above, there are teams actively looking into providing enhanced management, monitoring, and protection of Linux VMs (inc. WSL) and physical boxes, details of which will emerge over time.

If you’d like to enquire/discuss further, do reach out via email to crloewen and richturn at you know where dot com.

@colinmollenhour I don’t think anyone here is claiming that WSL2 isn’t the fix for this issue. We’re just pointing out that @lewisdonofrio is being insensitive by claiming that it’s fixed now because it just isn’t.

None of these suggestions worked for me unfortunately. I’m using “git status” to test and the same exact project which takes 0.4 seconds on my Ubuntu running within VirtualBox takes over 4.4 seconds on WSL. I’ve not disabled real time protection because that seems like a major security threat. So WSL is just about useless for real work, unfortunately.

I’ve tried the script and changed the .cargo entry to point to a toolchain I installed in my home directory but no such luck, AntiMalware still pins the CPU.

I can see things like cc1 and cc1plus up there in the task manager every now and then, but even at that I manually added the libexec directories containing binaries to the exclusions but with no luck.

After the series of updates (17120 -> 17128) I’m also facing both #3034 and this (Windows Defender high CPU load) issues. Even after full stack of exclusions I can’t run dev tools (tests for example, check attached pics please).

defender-cpu-load defender-exclusions

Just as a benchmark, I find turning off Windows Defender speeds up cp of a large directory by about a factor of 8.

Well, that the performance of the filesystem of WSL is slower than an actual system is a known issue since day 1 and well documented in benchmarks (e.g. Phoronix). The team even acknowledged that improving the performance is a high priority goal at the moment. The virus scanner performance impact also has always been there for me and if anything filesystem-related is happening it gets involved. McAfee is even worse than Defender. Disabling the Defender real-time protection helps but it’s still held back by basic performance limitations. I thought somebody once mentioned that it is a known issue caused by the way WSL is stored and it it would require reworking of it to get this fixed. So far it could live with it and had to move some more complex things I did on WSL to an real VM eventually because of other reasons so I hit me less. But since CLion is now having support for WSL and having no control over the virus scanner at all in an corporate environment makes this issue more severe for me now.

Note there are a few unrelated patterns that show poor performance on WSL. Turning off Defender will probably (possibly?) help for the many itty-bitty-files case

Erm, I don’t know if I buy this. Git status, commit and anything NPM related improve dramatically with Windows Defender disabled. I had to resort to force disabling via Group Policy to make any of these tools usable.

I can reproduce this very easily, doing a large cp with real-time protection turned on and off. Open and close calls in particular seem to take a long time, for example with protection then without protection:

10:15:49.752801 lstat("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000075>
10:15:49.752940 stat("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000063>
10:15:49.753057 open("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_RDONLY|O_NOFOLLOW) = 3 <0.003043>
10:15:49.756173 fstat(3, {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000042>
10:15:49.756358 open("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_WRONLY|O_TRUNC) = 4 <0.001099>
10:15:49.757559 fstat(4, {st_mode=S_IFREG|0777, st_size=0, ...}) = 0 <0.000071>
10:15:49.757727 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000059>
10:15:49.757878 read(3, "$ Grid indexing is 1-based\n\n$ 15"..., 131072) = 889 <0.000099>
10:15:49.758068 write(4, "$ Grid indexing is 1-based\n\n$ 15"..., 889) = 889 <0.000110>
10:15:49.758231 read(3, "", 131072)     = 0 <0.000071>
10:15:49.758356 close(4)                = 0 <0.000809>
10:15:49.759221 close(3)                = 0 <0.000045>

10:15:17.106272 lstat("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000061>
10:15:17.106379 stat("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000056>
10:15:17.106483 open("SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_RDONLY|O_NOFOLLOW) = 3 <0.000093>
10:15:17.106655 fstat(3, {st_mode=S_IFREG|0777, st_size=889, ...}) = 0 <0.000066>
10:15:17.106800 open("Q/SimpleTest/for_ian/stacscheck/params/Bombastic7_15.param.in", O_WRONLY|O_TRUNC) = 4 <0.000260>
10:15:17.107106 fstat(4, {st_mode=S_IFREG|0777, st_size=0, ...}) = 0 <0.000024>
10:15:17.107179 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000013>
10:15:17.107229 read(3, "$ Grid indexing is 1-based\n\n$ 15"..., 131072) = 889 <0.000034>
10:15:17.107301 write(4, "$ Grid indexing is 1-based\n\n$ 15"..., 889) = 889 <0.000041>
10:15:17.107381 read(3, "", 131072)     = 0 <0.000025>
10:15:17.107442 close(4)                = 0 <0.000077>
10:15:17.107554 close(3)                = 0 <0.000027>

Blaaaah this is killing me as projects get bigger. Has nobody at Microsoft looked at all? I’ve seen a ton of closed threads but no action. I have to disable Defender all the time to get Git to finish in reasonable amounts of time (2s for a checkout with it disabled vs over a minute). I’ve tried the workaround - it does nothing.