npgsql: Exception while reading from stream

Since the update from version 6.0.6 to 7.0.1, I have been seeing the same issue in the Testcontainers PostgreSQL module. I am not aware of this issue happens before. I have downgraded to the previous version and will monitor to see if it happens again.

Maybe this may help. I too have this issue and recently instrumented my app with Sentry. It captured two unhandled exceptions fo this error;

/_/src/Npgsql/Internal/NpgsqlReadBuffer.cs

System.IO.EndOfStreamException: Attempted to read past the end of the stream. File “NpgsqlReadBuffer.cs”, line 186, in async void NpgsqlReadBuffer.Ensure(int count)+EnsureLong(?) Npgsql.NpgsqlException: Exception while reading from stream File “NpgsqlReadBuffer.cs”, line 258, in async void NpgsqlReadBuffer.Ensure(int count)+EnsureLong(?) File “NpgsqlConnector.cs”, line 1301, in async IBackendMessage NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)+ReadMessageLong(?) x 2 File “NpgsqlDataReader.cs”, line 425, in async Task<bool> NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken) File “NpgsqlCommand.cs”, line 1414, in async ValueTask<NpgsqlDataReader> NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken) x 2 File “NpgsqlCommand.cs”, line 1217, in async ValueTask<object> NpgsqlCommand.ExecuteScalar(bool async, CancellationToken cancellationToken) File “/_/Dapper/SqlMapper.Async.cs”, line 1219, col 9, in async Task<T> SqlMapper.ExecuteScalarImplAsync<T>(IDbConnection cnn, CommandDefinition command)

AND:

/_/src/Npgsql/Internal/NpgsqlReadBuffer.cs

System.ObjectDisposedException: The CancellationTokenSource has been disposed. File “CancellationTokenSource.cs”, line 366, in void CancellationTokenSource.CancelAfter(uint millisecondsDelay) x 2 File “ResettableCancellationTokenSource.cs”, line 136, in void ResettableCancellationTokenSource.Stop() File “NpgsqlReadBuffer.cs”, line 207, in async void NpgsqlReadBuffer.Ensure(int count)+EnsureLong(?) File “NpgsqlConnector.cs”, line 1301, in async IBackendMessage NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)+ReadMessageLong(?) x 2 File “NpgsqlDataReader.cs”, line 425, in async Task<bool> NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken) File “NpgsqlCommand.cs”, line 1414, in async ValueTask<NpgsqlDataReader> NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken) x 2 File “NpgsqlCommand.cs”, line 1217, in async ValueTask<object> NpgsqlCommand.ExecuteScalar(bool async, CancellationToken cancellationToken) File “/_/Dapper/SqlMapper.Async.cs”, line 1219, col 9, in async Task<T> SqlMapper.ExecuteScalarImplAsync<T>(IDbConnection cnn, CommandDefinition command)

I found this comment in the code along the stack trace; // Most of the time, it should be fine to reset cancellation token source, so we can use it again // It’s still possible for cancellation token to cancel between reading and resetting (although highly improbable) // In this case, we consider it as timed out and fail with OperationCancelledException on next ReadAsync // Or we consider it not timed out if we have already read everything (count == 0) // In which case we reinitialize it on the next call to EnsureLong() if (async && count > 0) buffer.Cts.RestartTimeoutWithoutReset();

It really seems there is a bug here, likely with the async implementations .

On the surface it looks like a connection timeout issue, it however bubbles up as something that is confusing to everyone (this part seems very likely a bug on npgsql side). Really it should just say timeout so that we investigate other areas.

os: linux runtime: 6.0.13

Edit: for anyone this may help. For me the issue looks to certainly be timeouts. not caused by a slow query but the cost of opening a new connection ( i was closing aggressively and once i removed that code i am observing less of this). I found for example a span where it took a 1 second to open a connection and 30 milliseconds to execute the query.

In my case, it was AWS RDS Proxy resetting the connection as described here https://github.com/npgsql/npgsql/issues/3559

I have not seen this issue since implementing the fix.

@roji this should get reopened. Hiding behind a timeout error when there are obviously bugs when reading from stream (read the above messages from multiple people) is not right.

I promise you nobody’s hiding behind anything… But we have yet to receive any bug report we can actually look into. I understand that it’s especially difficult to reproduce this kind of bug, but it’s not impossible, and nobody has done that so far. We haven’t even seen a single packet dump that would indicate an issue on the Npgsql side.

On the other hand, timeouts really do occur because of network devices - I’ve seen that happen many times.

So I can open the issue, but I don’t see what we can do about it until someone provides the relevant info for us to look into…