aspnetcore: "The client reset the stream" stress errors?

ASP.NET Core 3.0.0-preview7.19365.7

I:

• downloaded the latest SDK from https://dotnetcli.blob.core.windows.net/dotnet/Sdk/master/dotnet-sdk-latest-win-x64.exe
• sync’d corefx to master
• cd d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress
• dotnet run -c Release

That runs HttpClient with HTTP/2 against Kestrel, using whatever SDK is installed.

When I do that, I get frequent failures like:

Error from iteration 1839 (GET Parameters) in task 5 with 19797 successes / 11 fails:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 627
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 655
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 700
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1698
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 1698
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPool.cs:line 541
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\RedirectHandler.cs:line 33
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpClient.cs:line 546
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\ClientOperations.cs:line 76
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_3>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\ClientOperations.cs:line 164
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass2_2.<<-ctor>g__RunWorker|4>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\StressClient.cs:line 147

and

Error from iteration 896 (POST Duplex Slow) in task 4 with 9115 successes / 10 fails:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The request was aborted.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
   at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Http.Http2Connection.FlushOutgoingBytesAsync() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Connection.cs:line 186
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 627
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 754
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http2Stream.cs:line 813
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionResponseContent.cs:line 47
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpContent.cs:line 436
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpContent.cs:line 443
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in D:\repos\corefx\src\System.Net.Http\src\System\Net\Http\HttpContent.cs:line 716
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\ClientOperations.cs:line 268
--- End of stack trace from previous location where exception was thrown ---
   at HttpStress.StressClient.<>c__DisplayClass2_2.<<-ctor>g__RunWorker|4>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\StressClient.cs:line 147

and if I instead run with dotnet run -c Release -aspnetlog in order to see server failures, I see errors like:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLOHSA333TBP", Request id "0HLOHSA333TBP:000045E3": An unhandled exception was thrown by the application.
System.IO.IOException: The request stream was aborted.
 ---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted.
 ---> System.IO.IOException: The read operation failed, see inner exception.
 ---> System.InvalidOperationException: Reading is already in progress.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_AlreadyReading()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.IO.Pipelines.StreamPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)
   at HttpStress.StressServer.<>c.<<MapRoutes>b__10_5>d.MoveNext() in d:\repos\corefx\src\System.Net.Http\tests\StressTests\HttpStress\StressServer.cs:line 172
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

in addition to ones (presumably expected due to cancellation) about the client resetting the stream.

If I instead run with dotnet run -c Release -httpSys, which uses UseHttpSys instead of UseKestrel, the stress app runs happily with no client errors until I eventually terminate it.

cc: @davidfowl, @halter73, @geoffkizer, @eiriktsarpalis