runtime: Connection reset by peer error when sending requests using HttpClient

Hi team,

So we’re having a .NET 6 app with Http client set up to send requests to an external API (which is an azure APIM behind which is also a .NET app) and around 1% < requests are failing due to the SocketException saying: “Unable to read data from the transport connection: Connection reset by peer. Connection reset by peer”.

Stack:

Inner exception System.IO.IOException handled at System.Net.Http.HttpConnection+<SendAsyncCore>d__64.MoveNext:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__186`1.MoveNext (System.Net.Security, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Security.SslStream+<ReadAsyncInternal>d__188`1.MoveNext (System.Net.Security, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnection+<<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|45_0>d.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnection+<SendAsyncCore>d__64.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

Inner exception System.Net.Sockets.SocketException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:

The result code in Azure shows “Faulted”, it usually takes only a few milliseconds and on the server side they couldn’t find traces of this requests reaching their end.

We haven’t changed any default timeouts on both ends and heres how we create the httpclient and send the requests on the client side:

public class HttpClientWrapper : IHttpClientWrapper
    {
        private readonly IHttpClientFactory httpClientFactory;

        public HttpClientWrapper(IHttpClientFactory httpClientFactory) =>
            this.httpClientFactory = httpClientFactory;

        public async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request)
        {
            using var client = httpClientFactory.CreateClient(CompressionConstants.HttpClientWithDecompressionName);
            return await client.SendAsync(request);
        }
    }

About this issue

Original URL
State: open
Created 7 months ago
Comments: 21 (13 by maintainers)

Most upvoted comments

you can also use my GH email.

wfurt on Dec 18, 2023

Sorry about that, can you please send it to miha.zupan instead?

MihaZupan on Dec 15, 2023

@wfurt Thank you! We implemented the retry pattern for this exception for starters as suggested by @CarnaViire and we’re monitoring it atm. We’ll try with the ExpectContinue header too to see if it helps preventing it. If it keeps happening, we’ll do a packet capture. I’ll keep you posted.

zspasojevic on Dec 6, 2023

I’m not 100% it would help e.g. this is experiment. The theory is like this: We have seen cases when server sends redirect or ask for authentication and then it would close connection. If this happens before whole request is sent HttpClient would throw. (and for POST the likelihood is somewhat large) The 100-continue allows server to send response (positive or negative) and HttpClient should deal with it without sending whole request body.

Packet capture would be definitive answer but I understand that this is difficult with the cloud. (not even talking about TLS). In general, you should not see reset IMHO as that is abnormal TCP closure. If this comes from the sever side (e.g. server or infrastructure) there is not much HttpClient can do. We just need more evidence to see what is actually happening - the exception itself is indication but it is not sufficient to find root cause.

wfurt on Dec 5, 2023