aspnetcore: Microsoft.AspNetCore.Hosting perf counters are slow

We’re trying to benchmark a webapp on a many-cores system and we’re mainly interested in RPS metrics at the moment. We don’t want to rely on client side RPS metrics (bombardier or Wrk depending on OS) because there are multiple clients and they’re not started instantly at the same time. It seems that Microsoft.AspNetCore.Hosting perf counters can be used to calculate RPS on aspnet side which is what we need. Unfortunately, it seems that these counters make everything twice slower when enabled.

Basically, it can be simulated on our PerfLab with crank:

crank 
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml 
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml 
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml 
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml 
--scenario json 
--profile arm-lin-28-app 
--profile intel-load2-load 
--application.framework net8.0 
--application.collectDependencies true 
--application.options.collectCounters true 
--application.options.counterProviders "Microsoft.AspNetCore.Hosting"

the last line is the culprit. Is it possible to somehow achieve an overhead-free server-side RPS metric? The current culprit seems to be EvenPipe:

Tried Linux-x64, Linux-arm64 and Windows-x64 machines.

UPD: Current suspects:

Mode	RPS
No counters	1M
Collect “System.Runtime” counters	1M
Collect “Microsoft.AspNetCore.Hosting” counters	0.48M
Collect “Microsoft.AspNetCore.Hosting” counters, EventPipeEnableStackwalk=0	0.65M
Collect “Microsoft.AspNetCore.Hosting” counters, EventPipeEnableStackwalk=0, without Interlocked	0.75M
Collect counters, Microsoft.AspNetCore.Hosting, EventPipeEnableStackwalk=0, without Interlocked, without lock in CounterAggregator	0.8M

About this issue

Original URL
State: open
Created 10 months ago
Comments: 17 (17 by maintainers)

Most upvoted comments

Are we fixing this for .NET 8?

davidfowl on Sep 2, 2023

There is a PR looking at it here: https://github.com/dotnet/aspnetcore/issues/50412

Did you mean https://github.com/dotnet/runtime/pull/91566?

noahfalk on Sep 7, 2023

Its possible to enable both, but there is likely no reason to do so. Looking at the code in ASP.NET Core, you may also be paying the costs to track both sets of metrics even if you only enabled one of them: source.dot.net/#Microsoft.AspNetCore.Hosting/Internal/HostingApplicationDiagnostics.cs,57 source.dot.net/#Microsoft.AspNetCore.Hosting/Internal/HostingApplicationDiagnostics.cs,353 Its possible to check if they are enabled individually, but currently the code doesn’t do that.

Adding separate checks for metrics vs event counters in ASP.NET Core hosting isn’t difficult. I’ll do that and aim for .NET 8. Edit: PR https://github.com/dotnet/aspnetcore/issues/50412

However, it won’t help if someone enables metrics and event counters. I don’t see why someone would want both. I think there is a task here to look at what .NET telemetry libraries and tooling are doing when they asks to listen for counters. I know dotnet-counters listens to both, but people aren’t going to use that in production, so it doesn’t matter. What about the .NET OpenTelemetry SDK? Or other counter gathering tooling?

I don’t think it is worth in investing in optimizing the HostingEventSource’s Interlocked.Increment calls. We should encourage people to move towards metrics.

JamesNK on Sep 7, 2023