aspnetcore: High CPU usage after await (with sample repo)

From @daddyman on July 19, 2018 21:47

Since moving to .NET Core 2.1 we have been seeing our aspnet core web api spike to 99% CPU usage on Linux. This does not happen when running on Windows.

What we observe in our application is if we make 4 or 5 API calls in quick succession then the CPU goes to 99% and stays there for a few minutes.

I was able to strip down our app and reproduce the issue in a simple app. The repo is here: https://github.com/daddyman/dotnetcorecpu

I am running .NET Core 2.1.2 using SDK 2.1.302 on Ubuntu 16.04.

It seems to have something to do with “await”. In the reproduction case I have one controller that handles a POST.

Performing an await Task.CompletedTask; does not cause a problem.
Performing an away DummyWait(); where DummyWait is return Task.Delay(1); causes high CPU after hitting the server with many requests in a short time. The requests complete quickly but then the CPU goes to 99% and stays there.

In our real app I did the same change of using the await Task.CompletedTask; and 'await Task.DummyWait();` and saw the same difference with the CPU usage.

Copied from original issue: dotnet/corefx#31208

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 17 (10 by maintainers)

Most upvoted comments

@Tratcher Doing UseLibuv() in our web host builder chain also fixed my problem.

beevik on Jul 21, 2018

@Tratcher, yes, we have similar problems

ZOXEXIVO on Jul 21, 2018

Tratcher on Jul 20, 2018

I’ve had a chance to repro this and try to understand the behavior a bit. What I see are after “enough” POST requests, Pairs of threads will saturate CPUs. Most commonly, I saw just a single pair (2) threads saturate two CPUs, but I did once encounter two pairs (4) threads saturate four CPUs.

Looking at the trace, the functions where the bulk of the CPU time are spent are:

Drilling into Pipe::ReadAsync shows that there is contention on this stack as well as a bunch of CPU time:

If I look at the time spent in managed code Monitors, it is coming from two methods (ReadAsync and AdvanceReader):

Note that this only seems to occur when I attempt to create contention by sending multiple requests at the same time, but the spike lasts for around a minute on my machine. The time spent in the spike seems to be attempting to process one or more requests and is slowly chipping away at them (with some contention).

It looks like this investigation should be continued by folks who have some familiarity with Pipelines.

brianrob on Jul 20, 2018