runtime: Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream')

Since ~last Sunday (July 18)~, every HTTP 1.1 Linux CI stress run has at least one occurrence of the following new failure type:

https://dev.azure.com/dnceng/public/_build/results?buildId=1248255&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=45b31e99-b18c-506f-46cf-7d3a4d5140a9

Stack Trace

System.Net.Http.HttpRequestException: An error occurred while sending the request.
client_1  |  ---> System.ObjectDisposedException: Cannot access a disposed object.
client_1  | Object name: 'SslStream'.
client_1  |    at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|138_0(ExceptionDispatchInfo e) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 849
client_1  |    at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 817
client_1  |    at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source, Boolean async) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 1447
client_1  |    at System.Net.Http.HttpConnection.FlushAsync(Boolean async) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 1422
client_1  |    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 503
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs:line 817
client_1  |    at System.Net.Http.HttpConnectionPool.SendUsingHttp11Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 981
client_1  |    at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1015
client_1  |    at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1028
client_1  |    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1038
client_1  |    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
client_1  |    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 553
client_1  |    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 95
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in /app/ClientOperations.cs:line 349
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 204

~After a quick naive look at recently merged System.Net.Http and System.Net.Security PR-s, #55772 seems to be the only one that matches the timing. Http PRs seem to be either test fixes, or unrelated to HTTP/ 1.1~

/cc @wfurt @Tratcher @geoffkizer @alnikola

Edit 1: Less frequently, but it also seems to happen on Windows.

Edit 2: First occurrence seems to be on 7/15

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 29 (28 by maintainers)

Commits related to this issue

Do not drain HttpContentReadStream if the connection is disposed (#57287) Fixes #56159. — committed to dotnet/runtime by antonfirsov 3 years ago

Most upvoted comments

@ManickaP @geoffkizer I managed to dig out more details. Cancellation isn’t happenning during HttpConnection.SendAsyncCore() but later, during a call to response.Content.LoadIntoBufferAsync() in HttpClient.SendAsync

Here are the series of events:

Response header has Transfer-Encoding: chunked, so HttpConnection.SendAsync assigns a ChunkedEncodingReadStream to response.Content
SendAsync returns
Since completionOption == HttpCompletionOption.ResponseContentRead, HttpClient.SendAsync(...) invokes response.Content.LoadIntoBufferAsync
This will eventually call into ChunkedEncodingReadStream.CopyToAsyncCore, which creates a CancellationTokenRegistration
The cancellation fires on a separate thread and kicks off Connection.Dispose
In the meanwhile ReadChunkFromConnectionBuffer calls _connection.CompleteResponse() which returns the disposed connection to the pool

antonfirsov on Aug 11, 2021

Hi @kedia990, yeah I think the cause of your problem will be very different, what we track here is a recent regression in 6.0 preview versions. Could you migrate your comment to a new issue? We can continue discussion from there. Also if you have a repro, it would be great if you shared it, so we can look at it a test it on our own.

ManickaP on Aug 11, 2021