AppMetrics: MetricsEndpointMiddleware doesn't satisfy AllowSynchronousIO option of Kestrel server

Recently we started migration to the prometheus engine to store our metrics, so we used the /metrics endpoint processed by this library. Also we have configured it with prometheus formatter.


    public class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost
                .CreateDefaultBuilder(args)
                .UseStartup<Startup>()
                .UseMetrics()
                .UseMetricsEndpoints(options =>
                {
                    options.MetricsEndpointOutputFormatter = new MetricsPrometheusTextOutputFormatter();
                })
                .UseKestrel(o => o.AllowSynchronousIO = false);
    }

There is an option in kestrel server which disable synchronous api in order to prevent potential blocking of threads. So what we observing is if AllowSynchronousIO == false, the formatters of library throw since they use synchronous methods when writing to the stream:

System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(String value)
   at App.Metrics.Formatters.Prometheus.Internal.AsciiFormatter.WriteSimpleValue(StreamWriter writer, String family, Double value, IEnumerable`1 labels, String namePostfix) in C:\projects\prometheus\src\App.Metrics.Formatters.Prometheus\Internal\AsciiFormatter.cs:line 115
   at App.Metrics.Formatters.Prometheus.Internal.AsciiFormatter.WriteMetric(StreamWriter streamWriter, MetricFamily family, Metric metric) in C:\projects\prometheus\src\App.Metrics.Formatters.Prometheus\Internal\AsciiFormatter.cs:line 110
   at App.Metrics.Formatters.Prometheus.Internal.AsciiFormatter.WriteFamily(StreamWriter streamWriter, MetricFamily metricFamily) in C:\projects\prometheus\src\App.Metrics.Formatters.Prometheus\Internal\AsciiFormatter.cs:line 58
   at App.Metrics.Formatters.Prometheus.Internal.AsciiFormatter.Write(Stream destination, IEnumerable`1 metrics, NewLineFormat newLine) in C:\projects\prometheus\src\App.Metrics.Formatters.Prometheus\Internal\AsciiFormatter.cs:line 36
   at App.Metrics.Formatters.Prometheus.MetricsPrometheusTextOutputFormatter.WriteAsync(Stream output, MetricsDataValueSource metricsData, CancellationToken cancellationToken) in C:\projects\prometheus\src\App.Metrics.Formatters.Prometheus\MetricsPrometheusTextOutputFormatter.cs:line 47
   at App.Metrics.AspNetCore.Endpoints.Middleware.MetricsEndpointMiddleware.Invoke(HttpContext context) in C:\projects\aspnetcore\src\App.Metrics.AspNetCore.Endpoints\Middleware\MetricsEndpointMiddleware.cs:line 41
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

As for now we have to set AllowSynchronousIO == true, but I think the library must be async all the way. The same behavior observed with default output formatter, and with /metrics-text default output formater. Checked with 2.0 and 3.0 versions of app.metrics, and with aspnetcore 2.1 and 2.2.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 15 (1 by maintainers)

Commits related to this issue

#396 #496 metrics text formatting async — committed to AppMetrics/AppMetrics by alhardy 5 years ago
#500 #496 #396 async updates and using system.text.json for core json formatting — committed to AppMetrics/AppMetrics by alhardy 5 years ago
#396 fix await async json serialization of metrics — committed to AppMetrics/AppMetrics by alhardy 5 years ago
#396 #396 removing allow sync io from sample apps — committed to AppMetrics/AppMetrics by alhardy 5 years ago
Features/4.0.0 (#503) * #494 Decommissiong health check functionality due to microsoft implementation * #496 bumping package versions for netcore3 * Features/buildupdate (#498) * #496 using ... — committed to AppMetrics/AppMetrics by alhardy 5 years ago
stable 4.0.0 release (#555) * updating app metrics version in global build ref * #494 Decommissiong health check functionality due to microsoft implementation * #496 bumping package versions fo... — committed to AppMetrics/AppMetrics by alhardy 4 years ago

Most upvoted comments

Any news about this? I think this one will become popular, as people are starting to migrate to core 3.0 now it’s officialy released.

hundreder on Oct 3, 2019

As @stukselbax said here: https://github.com/AppMetrics/AppMetrics/pull/507#issuecomment-581852762

There’s more work to be done, synchronous Write calls which need to be switched to WriteAsync etc

martinothamar on Feb 10, 2020

Going to start picking up core 3.0 related work, no ETA, any help appreciated, there’s a new features/4.0.0 branch

alhardy on Oct 25, 2019

It seems that Newtonsoft.Json.JsonWriter isn’t async, called from App.Metrics.Health.Formatters.Json.HealthStatusJsonOutputFormatter.WriteAsync :

InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.

Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(byte[] buffer, int offset, int count)
System.IO.StreamWriter.Flush(bool flushStream, bool flushEncoder)
System.IO.StreamWriter.Dispose(bool disposing)
System.IO.StreamWriter.Close()
Newtonsoft.Json.JsonTextWriter.CloseBufferAndWriter()
Newtonsoft.Json.JsonTextWriter.Close()
Newtonsoft.Json.JsonWriter.Dispose(bool disposing)
Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose()
App.Metrics.Health.Formatters.Json.HealthStatusJsonOutputFormatter.WriteAsync(Stream output, HealthStatus healthStatus, CancellationToken cancellationToken)
App.Metrics.AspNetCore.Health.Core.DefaultHealthResponseWriter.WriteAsync(HttpContext context, HealthStatus healthStatus, CancellationToken token)
App.Metrics.AspNetCore.Health.Endpoints.Middleware.HealthCheckEndpointMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
App.Metrics.AspNetCore.Tracking.Middleware.ApdexMiddleware.Invoke(HttpContext context)
App.Metrics.AspNetCore.Tracking.Middleware.PerRequestTimerMiddleware.Invoke(HttpContext context)
App.Metrics.AspNetCore.Tracking.Middleware.RequestTimerMiddleware.Invoke(HttpContext context)
App.Metrics.AspNetCore.Tracking.Middleware.ErrorRequestMeterMiddleware.Invoke(HttpContext context)
App.Metrics.AspNetCore.Tracking.Middleware.ActiveRequestCounterEndpointMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

pwlsjk on Oct 1, 2019

Just ran into this issue while migrating to .net core 3

razfriman on Sep 29, 2019

I know this is not the answer you’d like, but you should move away from OpenMetrics towards Opet Telemetry, at least for new projects.

hundreder on Oct 3, 2023