runtime: In .NET 7 large number of concurrent requests will memory leak

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

When TFM=.NET7 wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889 many times, memory will grow to 1G.

When TFM=.NET6 wrk -t8 -c1000 -d30s --latency http://192.168.2.86:8889 many times, memory only 145M.

The Code in program

            Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
            var builder = WebApplication.CreateBuilder(args);
            
            builder.Services.AddSingleton(new HitokotoManager(Init()));
            builder.Services.AddResponseCompression();

            var app = builder.Build();

            app.UseResponseCompression();
            app.Use(async (HttpContext builder, RequestDelegate next) =>
            {
                    //Write 1kb data in response
            });

Expected Behavior

Memory will not be much larger due to long concurrent requests

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

7.0.100

Anything else?

.NET SDK: Version: 7.0.100 Commit: e12b7af219

运行时环境: OS Name: Windows OS Version: 10.0.19045 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\7.0.100\

Host: Version: 7.0.0 Architecture: x64 Commit: d099f075e4

.NET SDKs installed: 3.1.426 [C:\Program Files\dotnet\sdk] 5.0.303 [C:\Program Files\dotnet\sdk] 5.0.403 [C:\Program Files\dotnet\sdk] 5.0.408 [C:\Program Files\dotnet\sdk] 6.0.112 [C:\Program Files\dotnet\sdk] 6.0.307 [C:\Program Files\dotnet\sdk] 7.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.0-preview.4.22251.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.0-preview.4.22229.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.19 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.30 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.31 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.0-preview.4.22229.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: arm64 [C:\Program Files\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation] x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

Learn more: https://aka.ms/dotnet/info

Download .NET: https://aka.ms/dotnet/download

About this issue

Original URL
State: open
Created a year ago
Comments: 33 (26 by maintainers)

Most upvoted comments

Ok, all the stuff about locks can be ignored. It is interesting, but not the root cause here.

The actual reason for the “leak” is simply because server GC gets really lazy for this scenario and does not collect very often. I see just one GC happening every few seconds and that sems to be Gen0.

There is no “leak” observed if workstation GC is used. (as in set DOTNET_gcServer=0)
It is also possible to limit the “leak” by forcing GC once in a while. Commit size stays within 20-30% above what CoreCLR uses, with something like:

        public static Thread gcForcer = new Thread(
                () =>
                {
                    for (; ; )
                    {
                        // force a GC once a while. It does not need to be blocking.
                        GC.Collect(2, GCCollectionMode.Forced, blocking: false);
                        Thread.Sleep(10000);
                    }
                }
             );

        public static async Task Main(string[] args)
        {
            gcForcer.Start();

In a way GC is not completely wrong here - there is plenty of RAM on the machine and it can afford 8Gb heap, but the expectations are clearly that it collects more often.
I do not think the NativeAOT runtime can change anything to fix this behavior.

I am not sure what changed in 8.0 that this scenario no longer reproduces (regardless of regions turned on or off). Perhaps GC tunnings have changed or something got fixed?

@mangod9 - I think GC team needs to take a look.

It is also possible that something changed in ASP.net/DependencyInjection and the app no longer meets the pattern that causes GC to be lazy.

Either way - something changed that caused GC to be less lazy and fixed this scenario in 8.0. Porting that change to 7.0, if possible, would be the likely fix.

VSadov on Aug 6, 2023

Also note that neither CoreCLR nor NativeAOT will create lock objects if they can get away with just using sync bits

For the perspective on this - with thin locks enabled this app has fat lock watermark at 8 on NativeAOT 8.0 And these locks seem to be long-lived as I do not see any recycling of entries in the steady state of the app.

Compared to 66K of simultaneous lock instances needed otherwise, thin locks is a quite impressive optimization.

VSadov on Aug 6, 2023

yeah its something in the runtime/nativeaot side I am guessing.

mangod9 on Jan 23, 2023