runtime: Idle application dies of OOM on Ubuntu 22.04 on .NET 7.0.2+

Description

We have a number of low-end instances running Ubuntu 22.04. They sport 2 vcpus and 0.5GB RAM. The application has been recently updated to .NET 7.0.2 and started some peculiar behavior.

On completely idle servers the managed memory metric starts to increase until the application is killed by OOM reaper.

Feb 27 13:30:02 ip-10-10-88-118 kernel: Out of memory: Killed process 498 (Raven.Server) total-vm:274342356kB, anon-rss:265784kB, file-rss:0kB, shmem-rss:7672kB, UID:1001 pgtables:3108kB oom_score_adj:0
Feb 27 13:30:01 ip-10-10-88-118 systemd[1]: raven.service: A process of this unit has been killed by the OOM killer.
...
Feb 27 13:30:02 ip-10-10-88-118 systemd[1]: raven.service: Main process exited, code=killed, status=9/KILL
Feb 27 13:30:02 ip-10-10-88-118 systemd[1]: raven.service: Failed with result 'oom-kill'.

It was not the case before on .NET 7.0.0 or 7.0.1.

After it’s killed most of the time the memory does not increase in such manner anymore, but application acts as normal and goes back to usual Managed memory usage. Some instances need to die at least twice to go back to normal behavior.

We determined that the memory is kept in GC, which for some reason ceases to run. Not sure why?

We have an endpoint to force GC cleanup and after calling it had quite a lot of memory cleaned up (for such an instance) e.g.:

{
	"BeforeGC": "358.6 MBytes",
	"AfterGC": "45.24 MBytes",
	"Freed": "313.36 MBytes",
	"DurationInMs": 807,
	"PinnedObjectsCount": 5,
	"FinalizationPendingCount": 1121,
	"PauseTimePercentage": "0.08%"
}

Afterwards the application continues to work as normal

When attempting dotnet-gcdump it immediately clears the GC contents freeing up the memory, so there’s no way for us to check what it’s been holding.

Configuration

Ubuntu 22.04 kernel 5.15.0-1030-aws 2 vcpus 0.5GB RAM

Regression?

Yes, since 7.0.2. We tried 7.0.3 as well - same thing. Worked OK before that version.

Data

gcdump done: