aspnetcore: Asp.Net Core API controller memory leak
Is there an existing issue for this?
- I have searched the existing issues
Describe the bug
We have identified an issue with Asp.Net core API controllers, specifically related to memory allocation which does not get cleared.
Our original use case was an API controller that reads a JSON file and returns data from it. We noted that memory use was growing every time we called the API controller.
We have generated a simplified example to reproduce this issue. Here, we use the following method to generate a 50 million character string and puts it into an object that is not used at all.
[HttpGet("[action]")]
public IActionResult Test1(int testNumber)
{
var result = new string((char)(new Random().NextInt64(1, 65535)), 50_000_000);
var mem = Process.GetCurrentProcess().PagedMemorySize64;
return Ok($"TEST #{testNumber} ; Mem: {mem:N0}");
}
However, the memory still seems to be allocated and retained for this object for a very long duration (GC is run but the memory doesn’t seem to be released). Interestingly, at some point, this hits a peak memory allocation, which differs based on the size of the string created.
The problem is also true when deserializing JSON files (which is what we started with) and returning a subset of information from the result object. In that case, the memory allocation for the complete object doesn’t seem to go away, causing a server to get overloaded.
In .Net 7, memory is never released, it seems. In .Net 6, some of the memory is released, some of the time, but we still don’t understand why it’s even stored in memory at all for any duration, since the object is not used for any purpose, and once the API call has completed, should be released.
You can test this issue with the attached solution. Once the project is run, monitor your Visual Studio’s Diagnostic Tools window (Process Memory). The API calls also return some information about memory, but the graph represents the issue better IMHO.
Expected Behavior
Objects are released from memory after method completes execution and the API response is returned
Steps To Reproduce
Run the TestWebAppCore project. Click on the links. Watch the memory not get released, but hit a top limit at some point per test.
Exceptions (if any)
No response
.NET Version
.Net 7.0.100 and .Net 6.0.400
Anything else?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 25 (8 by maintainers)
Show me the code snippet you use today and I’ll rewrite it for you.
By using streams. When you use the result types with objects, it doesn’t first buffer 50Mb into memory because they would be horrible for performance. Instead, it writes your 50Mb in 16K (configurable) chunks asynchronously to the response stream.
Hi @yarong-lifemap If i remember my .NET memory days correct (and .NET Core and Full Framework use roughly the same GC) then i think you problem is the size of the string you are allocating.
Any object bigger than 85.000 bytes will be allocated on the large object heap LOH
The garbage collection of LOH is different from the normal garbage collection in the sense that it only runs whenever .NET determines that its needed (eg. when you are running out of memory), this is because compacting memory consumes CPU cycles so if you constantly use the CPU to compact the memory then that CPU cannot also be used to serve requests (basically it’s a performance optimization).
I would guess that you can reproduce this memory leak in any version of .NET and .NET Core.
I’m not going to be the judge of “if you really need this massive size json (i would however recommend that you gzip (browsers support this out of the box) the response before returning it to your consumer)”.
Assuming that you need Mb size memory allocation to solve the problem you are facing I would recommend that you take a look at MemoryPool<T> which class that can help you do memory allocations and reuse of them which will move the GC responsibility from .NET Runtime into your code (before starting to use memory pooling i can recommend this article which highlights some of the pitfalls of the technique)
The memory leak problem is resolved in new update Version 17.4.1.
Thank you for contacting us. Due to a lack of activity on this discussion issue we’re closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn’t been addressed yet, please file a new issue.
This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!
@oferze I did a memory usage profile with the sample code you provided. I also added the
GetMemmethod you mentioned. Nothing seems abnormal to me.The 5 snapshots were taken at the beginning of the app and after a request to one of the endpoints in your code. As you can see, the memory was reclaimed when GC was triggered (taking snapshot will trigger a full GC).
Things here aren’t adding up, but if the issue is resolved then great 👍🏾.
I tested on .net7. Before update i observed the memory leak issue. I received visual studio update today afternoon. After updating i again tested and observed that garbage collector is removing unused items from memory. Also observed on graph diagnostic tool as well as in windows task manager. I haven’t tested in .net6
On Wed, 16 Nov, 2022, 8:18 pm InCerryGit, @.***> wrote:
I haven’t looked, but allocating a single 50MB/100MB/500MB string per request is a recipe for bad performance, see https://learn.microsoft.com/en-us/aspnet/core/performance/memory?view=aspnetcore-6.0.
Though this is bad, it should just make the GC run way more often. It’s likely that you are also running into this https://github.com/dotnet/aspnetcore/issues/45037.