runtime: WinHttpExceptions resulting in OutOfMemoryException

Possibly related to dotnet/runtime#22516

A few days ago our Azure website crashed because of an OutOfMemoryException. This happened during peak load of our application (.NET Core 1.2). Looking at the stdout logs, I saw a pattern where requests were starting to fail:

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.StreamToStreamCopy.<CopyAsyncAnyStreamToAnyStreamCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext()
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()

Eventually the dotnet process crashed, so our website was down. Looking at the logs the last entry was an OutOfMemoryException:

fail: Microsoft.AspNetCore.Server.Kestrel[0]
      UvAllocCb
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.MemoryPoolSlab.Create(Int32 length)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.MemoryPool.AllocateSlab()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Infrastructure.MemoryPool.Lease()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInput.IncomingStart()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection.OnAlloc(UvStreamHandle handle, Int32 suggestedSize)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Connection.<>c.<.cctor>b__47_1(UvStreamHandle handle, Int32 suggestedsize, Object state)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvStreamHandle.UvAllocCb(IntPtr handle, Int32 suggested_size, uv_buf_t& buf)
fail: Microsoft.AspNetCore.Server.Kestrel[0]
      Disposing listeners failed

I suspect this is not a coincidence and the two are somehow related. I was looking at how these WinHttpExceptions might cause an OutOfMemoryException. Only thing I can come up with, is not disposing of HttpRequestMessage and HttpResponseMessage. It seems there is a bit of confusion about this. Am I correct that in .NET Core 1.2 the HttpRequestMessage is disposed automatically after calling HttpClient.SendAsync? Or is there no relation between the exceptions and the OutOfMemoryException?

The other question is, what could cause the WinHttpExceptions? I’m quite sure we use the HttpClient correctly (we use a single instance for all requests). Only thing it does are REST calls to another Azure web app, also .NET Core 1.2. I am looking for a way to debug this properly.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 27 (11 by maintainers)

Most upvoted comments

For anyone interested (@zeiphon ?): the logger issue was caused by having stdout logs enabled in the web.config. We disabled this and timeouts are no longer occurring. This only happened during extremely high load, but it happened nonetheless.

razzemans on Dec 21, 2017

It’s nice because you get connection reuse, but you’re then capped at two concurrent connections/requests.

That is not true with .NET Core. The default number of concurrent connections per server is much higher than that (int.MaxValue).

And with .NET Framework, you can adjust that using ServicePointManager.DefaultConnectionLimit property.

davidsh on Oct 17, 2017

All the HttpClient.FinishSendAsync display classes on the heap lead me to believe that you’re running into threadpool starvation. You have a bunch of concurrent SendAsync calls, 400+ of which that are blocking threadpool threads because the call to @Html.Partial(MainSubNav... is not async.

This issue is exacerbated because the continuation in HttpClient.FinishSendAsync has very few or no threadpool threads to continue on and complete the call to SendAsync. So the HTTP request might very well be completing reasonably quickly, but the SendAsync might still be taking a very long time just because there are simply no threads to available complete on.

So the longer SendAsync takes, the less threads are available because more time is spent in Html.Partial. And the less threads that are available, the longer SendAsync takes because it has to wait longer to find a thread available for the FinishSendAsync continuation to run on. Simply put, it’s a vicious cycle very similar to the one I describe in this comment where I compare it to a deadlock.

I think changing the call to @await Html.PartialAsync(MainSubNav... might fix your entire problem.

halter73 on Oct 13, 2017

From the Razor perspective everything seems in place.

NTaylorMullen on Oct 13, 2017

@davidfowl You pinged the wrong Halter, Stephen is @halter73 😄

cesarblum on Oct 10, 2017