runtime: Error: Unable to read data from the transport connection: Connection reset by peer when using Httpclient in Dotnet core api

Description

Unable to read data from the transport connection: Connection reset by peer when using Httpclient in Dotnet core api

Error stacktrace:

   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Reproduction Steps

Running Dotnet core api in Docker hosted in AKS with HPA scaling, it’s an intermittent issue happens for 1-10/1000 Http calls to external api endpoints.

Followed Httpclient best practices.

Code Used:

DefaultHttpClientFactory httpClientFactory = new(null, new SocketsHttpHandler
        {
            PooledConnectionIdleTimeout = TimeSpan.FromMinutes(5),
            MaxConnectionsPerServer = 10
        });
        client = httpClientFactory.CreateClient("test");
        
   await client.SendAsync(request, cancellationToken.Token).ConfigureAwait(false);

Expected behavior

Successful http request with 200/error code

Actual behavior

Getting Error: Unable to read data from the transport connection: Connection reset by peer

Regression?

This was working in .Net Framework 4.7, from which we migrated to dotnet core 6 and deployed in AKS .

Known Workarounds

When we redeploy the service this error is reduced to some level for 8 hours and again it start to give the error intermittently

Configuration

.NET 6 Linux Ubuntu Kubernetes version: 1.25.0 Networking Azure CNI Nat gateway

Other information

No SNAT failures in NAT gateway and no Packet drops/Network errors in Grafana/Prometheus dashboard for the cluster level the endpoint accessed - is responding within milliseconds, but we get this error.