PowerShell: Windows Defender has started to slowdown method calls in PowerShell dramatically

We’ll pass this feedback to the Windows Defender team, but I would also suggest sending it via the Windows Feedback tool

The Windows Defender team has identified the issue and has a fix for this performance degradation, and the fix should be pushed by the end of May.

Defender has confirmed that this their issue and they are working on a fix.

I have been messing around with this further. The culprit is clearly the calls to AmsiNativeMethods.AmsiNotifyOperation. This call in and of itself is taking approximately ~40us to respond on each call, which is single-handedly causing the spike in latency. I have a version that I built which is caching the notifications sent to this method if they return the AMSI_RESULT_CLEAN result, as the documentation indicates that this return value should not change. I am caching with an LRU cache to only cache the last 500 calls (configurable) to limit the memory impact, though I am sure some logic between last recently used and most used would make more sense. This almost entirely mitigates the perf impact, and by placing the cache in the AmsiTools namespace, prevents manipulation of the cache in the same manner as the rest of the Amsi settings are protected. As I am unsure as to where the LRU cache implementation would be best located, I do not have a PR ready to submit for this, as I do think this entire topic needs further discussion to ensure a caching mechanism does not significantly increase the exposure of this very important security measure.

I had been chasing down a performance degredation in some of my existing modules/scripts over the last few weeks before I stumbled upon this issue. After doing a bit of digging, my issues seemed to closely align to this scenario so I wanted to take a closer look.

I performed a test using a local release build of 7.3.3 and then of the same tag with the AmsiNativeMethods.AmsiNotifyOperation removed (just setting its hr to 0) from the AmsiUtils.WinReportContent method. I then tested both builds with the following:

{
    $enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes(({
        $Count = 10000
        $str = 'A maasdlkfjoev 9r93 ldfjka.hdfhreu7\\ fkdjuenc'
        $SW = [Diagnostics.Stopwatch]::new()
        $SB = { [string]::new([char[]]$str.ToCharArray().ForEach({if([char]::IsLetterOrDigit($_)){return $_;}return ' '})) }
        $Total = 0l
        for ($i = 0; $i -lt $Count; $i++) {
            $SW.Restart()
            $null = $SB.InvokeReturnAsIs()
            $Total += $SW.ElapsedMilliseconds
        }
        $SW.Stop()
        $Total
        "# $($PSVersionTable['PSVersion']): { $($SB.ToString().Trim()) } took ${Total}ms for $Count iterations"
    }.ToString())))
    $Results = @("D:\Scratch\pwsh\7.3.3_no_amsinotify\pwsh.exe","D:\Scratch\pwsh\7.3.3\pwsh.exe").ForEach({ & $_ -NoProfile -NoLogo -EncodedCommand $enc })
    $Perf = ([long]$Results[2])/([double]([long]$Results[0]))
    Write-Host $Results[1]
    Write-Host $Results[3]
    Write-Host "Performance is $($Perf.ToString('F2')) times slower"
}.Invoke()

The version w/o AMSNofify ran 270x faster than the current release version:

# 7.3.3: { [string]::new([char[]]$str.ToCharArray().ForEach({if([char]::IsLetterOrDigit($_)){return $_;}return ' '})) } took 77ms for 10000 iterations
# 7.3.3: { [string]::new([char[]]$str.ToCharArray().ForEach({if([char]::IsLetterOrDigit($_)){return $_;}return ' '})) } took 20851ms for 10000 iterations
Performance is 270.79 times slower

This seems to indicate that indeed, the bottleneck is on the Amsi provider side (my test case was using Windows Defender as well)

I’m hoping we can get some official guidance on whether the remaining impact is considered acceptable and is the new normal for PowerShell 7.3+ going forward.

+1 I’ve been fighting this for a month or so now, and I also notice pwsh.exe hammering the Windows Defender registry key since this issue started.

I also have a test script that reproduces the issue. With RTP disabled, it takes 5 seconds, with RTP enabled, it takes 60 seconds.

# Set the number of iterations to perform
$iterations = 1000

# Start a timer to measure the time taken to execute the loop
$stopwatch = [System.Diagnostics.Stopwatch]::StartNew()

# Use a for loop to perform a series of calculations
for ($i = 1; $i -le $iterations; $i++) {
    # Perform some calculations here
    $result = 0
    for ($j = 1; $j -le $i; $j++) {
        $random = New-Object -TypeName System.Random
        $result += $random.Next(1,11)
    }

    # Update progress every 10 iterations
    if ($i % 10 -eq 0) {
        $percentComplete = ($i / $iterations) * 100
        Write-Progress -Activity "Calculating..." -PercentComplete $percentComplete -Status "Iteration $i of $iterations. Result = $result"
    }
}

# Stop the timer and calculate the elapsed time
$elapsedTime = $stopwatch.Elapsed
Write-Host "Elapsed time: $($elapsedTime.TotalSeconds) seconds"

This script will take ~40ms with realtime protection turned off, vs ~8000ms with realtime protection turned on. With realtime protection off: ~200ms, With realtime protection on: ~34000ms

I also run the script from https://github.com/PowerShell/PowerShell/issues/19431#issuecomment-1494528756 and see over 100x slowdown in execution time.

The AMSI method invocation logging was introduced in 7.3 and may play a role in this performance degradation. Reassign to the Security WG for evaluation and suggestions for next step.

I have the new version, but I’m still seeing the perf impact.

PS C:\> Get-MpComputerStatus | select am*

AMEngineVersion  : 1.1.23050.3
AMProductVersion : 4.18.23050.3
AMRunningMode    : Normal
AMServiceEnabled : True
AMServiceVersion : 4.18.23050.3

One of my scripts takes 0m:11s with RTP off (its original speed before all of this), but takes 1m:48s with RTP on. So nearly 10x slower. That is faster than it was when I first reported the issue, however, when it was around 50x slower.

That’s a real script too, btw, not repro code where I’m making useless method calls just to test the overhead.

@User1785604260 you can always close your own issues, but since a resolution label is set, our bot will close it automatically in a few days

@User1785604260,

would you also be willing to measure the time on my specific repro case from above?

Yes, for your test: With realtime protection on: ~34000ms With realtime protection off: ~200ms

I noticed the same behavior as I was busy trying to improve the performance of my script which appeared suddenly a lot slower. As these performance tests, I created recently, appeared now about 5 times slower…