runtime: Regression: Memory leak when launching child processes

A memory leak surfaced after upgrading our clients from .Net core 2.2 to 3.1. I’ve narrowed it (hopefully the only one) down with this example to have happened between 2.2 and 3.0:

for (int i = 0; i < int.MaxValue; i++) 
{
   using (var process = Process.Start("ls"))
       Console.WriteLine(i);
}

The above is memory stable targeting 2.2 but leaks on 3.0 and 3.1. Although it looks silly, I would consider it serious as it breaks any long running processes that cycle child processes.

Upon fixing, resources like process pipes should also be considered (see https://github.com/dotnet/runtime/issues/24476).

Configuration

Tested against 3.1.202 on macOS 10.15.3 (19D76) and linux-arm Raspbian GNU/Linux 8 (jessie)

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 36 (25 by maintainers)

Most upvoted comments

Ok, so that means that a growth in memory size by 150MB came from the native heap (malloc).

It is possible though that it is not a real leak, but rather a native heap fragmentation issue. What I mean is that say you allocate many blocks that sum together to 150MB by malloc, then you allocate say 100 bytes by malloc and then free all the 150MB blocks. And let’s assume that the 100 bytes were at the highest address in the heap. The heap cannot shrink after the free, as the heap is a continuous block of memory (IIRC).

The items you’ve got listed as “Memory not freed” by the mtrace could be those tiny bits preventing the heap from being able to shrink.

The document on how GLIBC malloc works supports this theory: https://sourceware.org/glibc/wiki/MallocInternals

Note that, in general, “freeing” memory does not actually return it to the operating system for other applications to use. The free() call marks a chunk of memory as “free to be reused” by the application, but from the operating system’s point of view, the memory still “belongs” to the application. However, if the top chunk in a heap - the portion adjacent to unmapped memory - becomes large enough, some of that memory may be unmapped and returned to the operating system

janvorli on May 28, 2020

@tmds thanks. I will now upgrade to the newest 3.1 version and do a test run with our devices.

mdisg on Oct 27, 2020

@mdisg I’ve added links to this issue and the PR that fixes it on https://github.com/dotnet/core/issues/3989.

tmds on Oct 27, 2020

@myrup ah yes. I created a request: https://github.com/dotnet/corefx/pull/42941

danmoseley on Jun 29, 2020

@danmosemsft Thanks! I’ll test it and report back here 👍

myrup on Jun 7, 2020

@tmds you could try to use mtrace or Valgrind to see if it detects native leaks from the native heap. Please note that you’ll see false positives as we don’t do any cleanup at exit. However, mtrace doesn’t track mmap allocated memory and I think that neither does Valgrind.

One more idea - let the process run until it accumulates a substantially large working set and then dump the /proc/PID/maps. That might help us to figure out where the memory comes from.

janvorli on May 21, 2020