reverse-proxy: Random 502 errors when client cancels request

Random 502 errors when ui cancels the request.

We have an Angular UI application that talks to an ASP.NET webservice hosted in IIS and proxyed with YARP. The backend only works with HTTP1.1 not HTTP/2 The error happens only if YARP uses HTTP/2, so the error happens only when YARP needs to proxy HTTP/2 traffic to a HTTP1.1 backend. The application works as usual and we have some custom ui logic that cancels some of the requests, when the requests is cancelled YARP returns 502.

To Reproduce

I had not created a reproducible working scenario but I think that it is enough to issue a request then immediately cancels. From the browser I can see that the request has content-lenght of some bytes (49 in the above example) but the request was cancelled before the first byte of the content was transmitted so YARP complained.

2023-03-14 14:52:36.832 +01:00 [INF] Proxying to http://localhost/api/v1/omnisearch/view-config HTTP/2 RequestVersionOrLower no-streaming
2023-03-14 14:52:36.836 +01:00 [INF] "Request": An error was encountered before receiving a response.
System.Net.Http.HttpRequestException: Sent 0 request content bytes, but Content-Length promised 49.
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)

Further technical details

Latest YARP packages, on .NET 7 windows environment.

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 17 (7 by maintainers)

Most upvoted comments

Sorry I merged it a bit fast, it should be ok now

benjaminpetit on Apr 28, 2023

Sorry for not coming back sooner.

If I do launch the demo with dotnet run --project ... I see the exception. If I run the dlls directly with dotnet xxx.dll the repro doesn’t work (not sure why I tried first that way).

We should be able to do a workaround for this issue soon.

Thanks again for the repro.

benjaminpetit on Apr 19, 2023

Hello @MihaZupan Hello @alkampfergit

We have encountered the same problem in our environment. The problem has been found with update to Yarp 2.0.

But here with a Blazor Wasm client, SingalR and GRPC. The problem seems to occur only with Kestrel. In an IIS environment the problem seems to occur less often.

Attached you can find a sample application consisting of a WASM client which opens a SignalR connection and then makes a GRPC call. Then the SignalR connection is closed and another GRPC call is made. This is done in a loop until the error occurs. The problem can be reproduced more quickly in Chrome than in Firefox. ProblemDemo.zip

With Yarp, the error occurs after about 120 runs:

System.IO.IOException: The client reset the request stream.
         at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
         at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer)
fail: Grpc.AspNetCore.Server.ServerCallHandler[6]
      Error when executing service method 'SampleCall'.
      System.IO.IOException: The client reset the request stream.
         at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
         at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
         at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall)

ClientError

Without Yarp the whole thing runs stable: NoError

What’s interesting is that the client can’t make any other calls to the server at all after the error. The error occurs then immediately. If you wait about 30 seconds, the system recovers and several calls are possible again.

Schaeri on Mar 21, 2023

the ui cancels the request and the user has no visual indication that the query was canceled Since from angular point of view the call is not canceled

How is it canceled, but not considered canceled? Can you share a runnable repro js script we could use that demonstrates that?

I really like this kind of message to be logged at least in WARN, am I wrong?

You’re in luck, this was just recently changed to warning in main: #2044

MihaZupan on Mar 15, 2023