aspnetcore: Kestrel dies because of Out of Memory exception
One of our MVC projects keeps growing memory, and it seems that Garbage collector dose not do anything with it.
The process keeps growing until it is killed by the OS.
dmesg outputs this log:
[961030.365411] Out of memory: Kill process 3673 (dotnet) score 913 or sacrifice child
[961030.365725] Killed process 3673 (dotnet) total-vm:29592364kB, anon-rss:3565416kB, file-rss:256kB, shmem-rss:0kB
What makes me think that it is related to, GC not working as expected is that, i have periodically called GC.Collect() and memory immediately drops down significantly but still keeps growing in a much slower pace.
Also GC is running under Server mode.
Environment Info
Runtime: .Net Core 2.1
ASP.NET Core 2.1
Web server: Kestrel (directly exposed)
OS: Centos 7
4 Core processor
4GB Memory
Also not sure if it any how related, i thought maybe GC dose not recognize memory limits, so here is the output of ulimit command:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15062
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15062
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Notes
- the apps works as expected on Windows server 2016.
- this may or may not be a duplicate of #1976 but all the issues in there are being tracked separately.
Update
Adding entire dmesg log here as gist to keep things clear.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 68 (66 by maintainers)
@benaadams I have tested this locally under Profiler and it was completely fixed. 👍 Will try it in production later this week.
Thanks @benaadams @davidfowl
We addressed it in 2.2, we can consider patching this in 2.1 but it might be too big of a change to do that. I’ll discuss it with some people.
cc @muratg @Eilon @DamianEdwards
This wasn’t a single issue, it was a whole category of issues. Several linked work items were created and some of those have already been fixed in the 2.2.0 previews. https://github.com/aspnet/Mvc/issues/8362 looks most directly related to your last post.