SqlClient: System.IO.IOException: Unable to write data to the transport connection: Broken pipe.

Getting the same error running the following:

Netcoreapp2.2 / Linux container (mcr.microsoft.com/dotnet/core/runtime:2.2) / Kubernetes 0.13.5 / SQL Azure ( Elastic Pools, Standard Plan )

Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Data.SqlClient.SNI.SslOverTdsStream.WriteInternal(Byte[] buffer, Int32 offset, Int32 count, CancellationToken token, Boolean async)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStreamInternal.WriteSingleChunk[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStreamInternal.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsConnection.Send(SNIPacket packet)
   at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)
   at System.Data.SqlClient.SNI.SNIMarsHandle.Dispose()
   at System.Data.SqlClient.SNI.TdsParserStateObjectManaged.Dispose()
   at System.Data.SqlClient.TdsParserSessionPool.Dispose()
   at System.Data.SqlClient.TdsParser.Disconnect()
   at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()
   at System.Data.ProviderBase.DbConnectionPool.DestroyObject(DbConnectionInternal obj)
   at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

This causes the pods to restart with a 139 exit code, and we can’t prevent it 😞

Currently, this is a show stopper for us, are there any workarounds?

I noticed a very suspicious throw introduced long ago that would easily cause this issue if disposing a broken, mars-enabled connection: https://github.com/dotnet/corefx/pull/4594/files#diff-a8e6146000f306b97ce4e48207f7cd9fR69 I plan to remove the throw for the next M.D.SqlClient preview.

The stack trace is saying a lot:

A timeout occurs:

identity-api         |    at System.Threading.TimerQueue.FireNextTimers()

It cleans up a connection from the connection pool:

identity-api         |    at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)

The connection gets disposed:

identity-api         |    at System.Data.SqlClient.SqlInternalConnectionTds.Dispose()

A control packet is sent through the connection:

identity-api         |    at System.Data.SqlClient.SNI.SNIMarsHandle.SendControlPacket(SNISMUXFlags flags)

But the connection was already closed:

identity-api         |    Unhandled Exception: System.AggregateException: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.) ---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe. ---> System.Net.Sockets.SocketException: Broken pipe

@AfsanehR, this seems fairly problematic if it’s actually happening frequently for multiple apps. I realize it could be challenging to address without a repro, but I’ve moved it from Future to 3.0 for now so that it doesn’t slip through the cracks. cc: @karelz

Your not supposed to catch this one, corefx is.

Hi there, I’m having the same issue. I’m using Microsoft.EntityFrameworkCore.SqlServer version 2.2.1 already. Any other clues on how to fix this issue?

Nice one, thanks @David-Engel !

@pawelpabich @mifurm @droyad @johnsimons Is it possible to provide a link to repro app to reproduce the issue locally and study traces?

I agree that the application should not exit leading to Pod restart if the trigger is this IO exception. I can take a deeper look with user-scenario handy, and then provide some insights.