csharp: What is the possible cause of "Connection reset by peer" when watching ListNamespacedPodWithHttpMessagesAsync

Hi!

A have an AKS (Azure Kubernetes Service) and recently when I deployed my app I started receiving an exception:

System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
   at k8s.WatcherDelegatingHandler.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal()
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)

   ...

Later it stopped responding like that. Is it a transient error? Or should I investigate?

P.S. My app is based on Microsoft.Orleans and there is a Orleans.Hosting.Kubernetes extension which uses ListNamespacedPodWithHttpMessagesAsync API.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (15 by maintainers)

Most upvoted comments

The 7.0.13 fixed Connection reset by peer in my test, @AntonPetrov83 you can take a try. The .net version should be 5.0+.

Connection reset by peer might be caused by any routing node to the dst endpoint. it is a underlayer network issue.

in the case #533, we noticed that it caused by LB kicked idle connections. retrying maybe the solution if it is caused by network layer.