OrchardCore: Struggling with high memory usage

Describe the bug

My scenario: We have a site built on Orchard Core 1.6. We have a pipeline that periodically deploys an instance to a test server and runs a suite of UI tests against it (the tests are using MSTest and Playwright, if it makes any difference). The site is running in IIS. The test suite runs for about 1 hour, and by the end of that run the w3wp process is consuming over 8.5 GB. ASPNETCORE_ENVIRONMENT is set to “Development”.

Here’s a quick look at the memory usage over time. This is the memory of the entire machine, not just OC, but looking in Task Manager, I see w3wp using over 8.5 GB.

To Reproduce

I’ve been struggling for quite a while on pinpointing the root issue, with no success thus far. I’ve tried using the Visual Studio memory profiler but I’ve been struggling to make any sense of it.

I next tried a much simpler example. This next example is a plain old OC site (latest from the main branch) using the blog recipe. That’s it. ASPNETCORE_ENVIRONMENT is set to “Development”. I then wrote a small PowerShell script to ping the site over and over.

function Invoke-Path($path) {
    Invoke-WebRequest -Uri "https://localhost:8001/${path}"
}

$Paths = @(
    '', 
    'categories', 
    'tags', 
    'Contents/ContentItems/40rg5j34w43er1a5dwg0j14cnr', 
    'login'
)

while (1) {
    $Paths | ForEach-Object -Process {
        Invoke-Path $_
    }
}

I ran this script for a little while. Looking at the diagnostic tools in Visual Studio, I can see multiple garbage collector invocations (the yellow arrows), but the memory never seems to decrease. I was hoping to see a more jagged pattern in the memory usage.

At this point I’m looking for any ideas or advice. Thanks.

About this issue

Original URL
State: closed
Created a year ago
Comments: 76 (76 by maintainers)

Most upvoted comments

@MikeAlhayek

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-7.0#httpclient-and-lifetime-management

rjpowers10 on Sep 21, 2023

ThrillerMichaelJacksonGIF

sebastienros on Sep 19, 2023

@rjpowers10 @ShaneCourtrille @MikeAlhayek @wAsnk @sebastienros

I saw one of the last meeting related to string allocations, interesting. For info by using instance ids under the debugger we can see that compiled items loaded from assemblies hold different instances of the "mvc.1.0.view" string.

Razor string duplicates

In #14348 we now override how compiled items are loaded and we can see that the same instance is used.

Razor string no duplicate

Also I could reduce the number of times the compiled items are loaded, before 3 times per tenant building, for example the view descriptors are populated on demand by models providers (one from aspnetcore, one from orchardcore), now only once by caching the descriptors built in a given shell scope.

We can still have duplicate strings but they are no longer rooted and can be reclaimed, the only one holding these strings being our static compiler that we now always use, and now it only references one instance of "mvc.1.0.view".

jtkech on Sep 24, 2023

Okay fixed locally but need to be tweaked, I will update the PR soon.

jtkech on Sep 19, 2023

@rjpowers10

Good to know, thanks.

Yes, Releasing a tenant was not a problem in itself, the first issue was related to HttpClient.
But Reloading a tenant where we rebuild the Configuration stack still has memory leaks. Normally we only reload a tenant on setup or when enabling / disabling features, not when updating settings, so if you have time you could retry your tests with all the features already enabled, if possible.

I will work on this part related to tenant Reloading asap, maybe a separate PR.
About razor strings I may have some ideas to reduce the number but I don’t think they are rooted.

jtkech on Sep 18, 2023

@rjpowers10 for info

Okay I could override DefaultHttpClientFactory, made it IDisposable to nullify some fields, for example _services, for now it fixes the issue if we set the options .SuppressHandlerScope to true, otherwise still some memory increase (but much less).

I also started to prevent the accumulation of timer callbacks that cleanup the handlers after their lifetime, but the lifetime can be higher than the period we use to release the tenant, but I made some progress.

jtkech on Sep 16, 2023

Okay, fixed by #14499

jtkech on Oct 12, 2023

@rjpowers10 @ShaneCourtrille and others.

Here a little summary of what has been done done in #14348, if you can give it a try.

So in #14348 we override the IHttpClientFactory, mainly to make it IDisposable so that when releasing / reloading a Tenant we release clients Handlers and cleanup Timers.
On Tenant Reloading, we now also Release the 2 inner IConfiguration of ShellSettings.
Finally we now always use our static shared SharedViewCompilerProvider (see below).

About razor view compiled items, we already have a static shared compiler provider but that we only use if razor runtime compilation is enabled, now in #14348 we always register it.

        // Shares across tenants a static compiler even if there is no runtime compilation
        // because the compiler still uses its internal cache to retrieve compiled items.
        services.AddSingleton<IViewCompilerProvider, SharedViewCompilerProvider>();

We still load the CompiledDescriptor from assemblies but not to be hold by another compiler instance, the cache of the same static compiler is populated once.

Doing so it seems that the GC behavior is better, here after many Tenant Release / Reload.

Memory Leaks

Memory GC

jtkech on Sep 22, 2023

@rjpowers10 I’m still not done looking but my initial checks look good for the mvc.1.0.view fix and in our case that saves us 500MB of memory alone. /cc @jtkech (FYI)

ShaneCourtrille on Sep 19, 2023

Okay, I may have found the source of the 2nd problem related to tenant Reloading where we rebuild the Configuration stack using some providers inheriting from FileConfigurationProvider.

See https://github.com/dotnet/runtime/issues/86146 and https://github.com/dotnet/runtime/pull/86455

jtkech on Sep 19, 2023

@rjpowers10 Out of curiosity… have you taken a look at your strings? I’m doing some analysis and found an oddity where an estimated 80% of strings that are 46 bytes long are the value “mvc.1.0.view” being repeated over and over. What’s really painful is that these strings (or at least the ones I’ve been able to sample since gcroot is such an expensive operation) don’t have a gcroot but are just sort of sitting around.

There are a ton of strings in my memory dump, although only four instances of “mvc.1.0.view”.

EDIT: Whoops, read the memory dump wrong. I have over 260,000 occurrences of “mvc.1.0.view”.

rjpowers10 on Sep 18, 2023

RouteEndpoints in the diff would be explained by a tenant being loaded.

Coming back to your original description, the issue is that your test is taking 8.5Gb, can you try to do the same thing with ASPNETCORE_ENVIRONMENT in Production, not “Development”, just to see if that has an impact.

Can you take a memory dump when the app is close to the end to see what is still kept in memory?

sebastienros on Aug 17, 2023

@MikeAlhayek thanks for the suggestion but in both my examples (my application built on OC and a plain old OC blog) the cache setting is set to “from environment”. ASPNETCORE_ENVIRONMENT is set to ‘Development’ for both, so the dynamic cache should be disabled. I tried forcing the setting to ‘Disabled’ as a sanity check and the results are the same.

rjpowers10 on Aug 14, 2023