npgsql: Random Unknown message code: 0-255 without SSL (data corruption)

Corruption between the npgsql<-postgres reading. Something is very badly bugged and this will be difficult to diagnose.

Steps to reproduce

This only seems to happen in a loaded production environment and happens very rarely/randomly with any EF command irregardless of what it’s actually doing or how it’s injected.

The other issues related to this use SSL, I am not using SSL.

AppDatabaseContext:

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseNpgsql(
                $"Server={server};Port={port};Database={db};User Id={user};Password = {pass};Timeout=120;CommandTimeout=120");
            optionsBuilder.UseLazyLoadingProxies();
            optionsBuilder.EnableSensitiveDataLogging();
        }

Startup.cs:

services.AddDbContext<AppDatabaseContext>(ServiceLifetime.Transient);

Inject into a controller, razor page, etc, the usual way you would with DI (in the controller’s constructor for example, or @inject in razer page), perform any async query e.g. var settings = await Db.SomeTable.FirstOrDefaultAsync(w => w.id == 1);

The issue

Randomly there will be corruption reading the response from the database server. This will cause postgresql to create ghost sessions that block maintenance activities, stuck in ClientRead with no actual process running on the system, until the ASP.NET app is restarted: Cancelling the task (or most likely - long running (1-5 second) queries - as these are the ones getting cancelled if the user quits the session) seems to cause this more often but it’s happening with anything, even basic queries like the one above.

Some examples:

Npgsql.NpgsqlException: Unknown message code: 161
  ?, in void PGUtil.ValidateBackendMessageCode(BackendMessageCode code)
  ?, in async IBackendMessage NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)+ReadMessageLong(?) x 2
  ?, in async Task<bool> NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken)
  ?, in async ValueTask<NpgsqlDataReader> NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken) x 2
  ?, in async Task<DbDataReader> NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
  File "RelationalCommand.cs", in async Task<RelationalDataReader> RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) x 2
  File "SingleQueryingEnumerable.cs", in async Task<bool> AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
  ?, in async Task<TResult> NpgsqlExecutionStrategy.ExecuteAsync<TState, TResult>(TState state, Func<DbContext, TState, CancellationToken, Task<TResult>> operation, Func<DbContext, TState, CancellationToken, Task<ExecutionResult<TResult>>> verifySucceeded, Canc...
  File "SingleQueryingEnumerable.cs", in async ValueTask<bool> AsyncEnumerator.MoveNextAsync()
  File "ShapedQueryCompilingExpressionVisitor.cs", in async Task<TSource> ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync<TSource>(IAsyncEnumerable<TSource> asyncEnumerable, CancellationToken cancellationToken) x 2
  File "Controllers/TranslateAPIController.cs", line 454, col 13, in async Task<IActionResult> TranslateAPIController.VersionAsync(int verCode)
  File "ActionMethodExecutor.cs", in async ValueTask<IActionResult> TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, object controller, object[] arguments)
  File "ControllerActionInvoker.cs", in async Task ControllerActionInvoker.InvokeActionMethodAsync()+Awaited(?)
  File "ControllerActionInvoker.cs", in async Task ControllerActionInvoker.InvokeNextActionFilterAsync()+Awaited(?)
  File "ControllerActionInvoker.cs", line 517, in void ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
  File "ControllerActionInvoker.cs", line 289, in Task ControllerActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
  File "ControllerActionInvoker.cs", line 494, in Task ControllerActionInvoker.InvokeInnerFilterAsync()
  File "ResourceInvoker.cs", in async Task ResourceInvoker.InvokeNextResourceFilter()+Awaited(?)
  File "ResourceInvoker.cs", line 1465, in void ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
  File "ResourceInvoker.cs", line 903, in Task ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
  File "ResourceInvoker.cs", line 260, in Task ResourceInvoker.InvokeFilterPipelineAsync()
  File "ResourceInvoker.cs", in async Task ResourceInvoker.InvokeAsync()+Logged(?) x 2
  File "EndpointMiddleware.cs", in async Task EndpointMiddleware.Invoke(HttpContext httpContext)+AwaitRequestTask(?)
  File "AuthorizationMiddleware.cs", in async Task AuthorizationMiddleware.Invoke(HttpContext context)
  File "AuthenticationMiddleware.cs", in async Task AuthenticationMiddleware.Invoke(HttpContext context)
  File "/_/src/Sentry.AspNetCore/SentryTracingMiddleware.cs", line 246, col 5, in async Task SentryTracingMiddleware.InvokeAsync(HttpContext context) x 2
  File "StatusCodePagesMiddleware.cs", in async Task StatusCodePagesMiddleware.Invoke(HttpContext context)
  File "ExceptionHandlerMiddleware.cs", in async Task ExceptionHandlerMiddleware.Invoke(HttpContext context)+Awaited(?)

Npgsql.NpgsqlException: Unknown message code: 227
  ?, in void PGUtil.ValidateBackendMessageCode(BackendMessageCode code)
  ?, in async IBackendMessage NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)+ReadMessageLong(?) x 2
  ?, in async Task<bool> NpgsqlDataReader.NextResult(bool async, bool isConsuming, CancellationToken cancellationToken)
  ?, in bool NpgsqlDataReader.NextResult()
  ?, in async ValueTask<NpgsqlDataReader> NpgsqlCommand.ExecuteReader(CommandBehavior behavior, bool async, CancellationToken cancellationToken) x 2
  ?, in NpgsqlDataReader NpgsqlCommand.ExecuteReader(CommandBehavior behavior)
  File "RelationalCommand.cs", line 541, in RelationalDataReader RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
  File "SingleQueryingEnumerable.cs", line 243, in bool Enumerator.InitializeReader(Enumerator enumerator)
  ?, in TResult NpgsqlExecutionStrategy.Execute<TState, TResult>(TState state, Func<DbContext, TState, TResult> operation, Func<DbContext, TState, ExecutionResult<TResult>> verifySucceeded)
  File "SingleQueryingEnumerable.cs", line 162, in bool Enumerator.MoveNext()
  File "EntityFrameworkQueryableExtensions.cs", line 3099, in void EntityFrameworkQueryableExtensions.Load<TSource>(IQueryable<TSource> source)
  File "EntityFinder.cs", line 125, in void EntityFinder<TEntity>.Load(INavigation navigation, InternalEntityEntry entry)
  File "LazyLoader.cs", line 108, in void LazyLoader.Load(object entity, string navigationName)
  File "LazyLoadingInterceptor.cs", line 51, in void LazyLoadingInterceptor.Intercept(IInvocation invocation)
  ?, in void AbstractInvocation.Proceed()
  ?, in FriendServantModel FriendDeckModelProxy.get_servant()
  File "Components/SupportDeckItem.razor", line 59, col 9, in void SupportDeckItem.BuildRenderTree(RenderTreeBuilder __builder)
  File "ComponentState.cs", line 97, in void ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment, out Exception renderFragmentException)
  File "HtmlRenderer.cs", line 70, in void HtmlRenderer.HandleException(Exception exception)
  File "Renderer.cs", line 970, in void Renderer.HandleExceptionViaErrorBoundary(Exception error, ComponentState errorSourceOrNull)
  File "Renderer.cs", line 659, in void Renderer.ProcessRenderQueue()
  File "HtmlRenderer.cs", line 70, in void HtmlRenderer.HandleException(Exception exception)
  File "Renderer.cs", line 659, in void Renderer.ProcessRenderQueue()
  File "Renderer.cs", line 513, in void Renderer.AddToRenderQueue(int componentId, RenderFragment renderFragment)
  File "ComponentBase.cs", line 116, in void ComponentBase.StateHasChanged()
  File "ComponentBase.cs", line 276, in Task ComponentBase.CallOnParametersSetAsync()
  File "ComponentBase.cs", in async Task ComponentBase.RunInitAndSetParametersAsync()
  File "HtmlRenderer.cs", line 70, in void HtmlRenderer.HandleException(Exception exception)
  File "Renderer.cs", line 970, in void Renderer.HandleExceptionViaErrorBoundary(Exception error, ComponentState errorSourceOrNull)
  File "Renderer.cs", in async Task Renderer.RenderRootComponentAsync(int componentId, ParameterView initialParameters)
  File "HtmlRenderer.cs", in async Task<ValueTuple<int, ArrayRange<RenderTreeFrame>>> HtmlRenderer.CreateInitialRenderAsync(Type componentType, ParameterView initialParameters)
  File "HtmlRenderer.cs", in async Task<ComponentRenderedText> HtmlRenderer.RenderComponentAsync(Type componentType, ParameterView initialParameters)
  File "RendererSynchronizationContext.cs", in async Task RendererSynchronizationContext.InvokeAsync(Action action)+(?) => { }
  File "StaticComponentRenderer.cs", in async Task<IEnumerable<string>> StaticComponentRenderer.PrerenderComponentAsync(ParameterView parameters, HttpContext httpContext, Type componentType)
  File "ComponentRenderer.cs", in async Task<IHtmlContent> ComponentRenderer.StaticComponentAsync(HttpContext context, Type type, ParameterView parametersCollection)
  File "ComponentRenderer.cs", in async Task<IHtmlContent> ComponentRenderer.RenderComponentAsync(ViewContext viewContext, Type componentType, RenderMode renderMode, object parameters)
  File "Areas/Support/Pages/Support.cshtml", line 476, col 27, in async Task Areas_Support_Pages_Support.ExecuteAsync()
  File "RazorView.cs", in async Task RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
  File "RazorView.cs", in async Task<ViewBufferTextWriter> RazorView.RenderPageAsync(IRazorPage page, ViewContext context, bool invokeViewStarts)
  File "RazorView.cs", in async Task RazorView.RenderAsync(ViewContext context)
  File "ViewExecutor.cs", in async Task ViewExecutor.ExecuteAsync(ViewContext viewContext, string contentType, int? statusCode) x 2
  File "ResourceInvoker.cs", in async Task ResourceInvoker.InvokeNextResultFilterAsync<TFilter, TFilterAsync>()+Awaited(?)
  File "ResourceInvoker.cs", line 1511, in void ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
  File "ResourceInvoker.cs", line 1343, in Task ResourceInvoker.ResultNext<TFilter, TFilterAsync>(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
  File "ResourceInvoker.cs", line 1120, in Task ResourceInvoker.InvokeResultFilters()
  File "ResourceInvoker.cs", in async Task ResourceInvoker.InvokeNextResourceFilter()+Awaited(?)
  File "ResourceInvoker.cs", line 1465, in void ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
  File "ResourceInvoker.cs", line 903, in Task ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
  File "ResourceInvoker.cs", in async Task ResourceInvoker.InvokeFilterPipelineAsync()+Awaited(?)
  File "ResourceInvoker.cs", in async Task ResourceInvoker.InvokeAsync()+Logged(?) x 2
  File "EndpointMiddleware.cs", in async Task EndpointMiddleware.Invoke(HttpContext httpContext)+AwaitRequestTask(?)
  File "AuthorizationMiddleware.cs", in async Task AuthorizationMiddleware.Invoke(HttpContext context)
  File "AuthenticationMiddleware.cs", in async Task AuthenticationMiddleware.Invoke(HttpContext context)
  File "/_/src/Sentry.AspNetCore/SentryTracingMiddleware.cs", line 246, col 5, in async Task SentryTracingMiddleware.InvokeAsync(HttpContext context) x 2
  File "StatusCodePagesMiddleware.cs", in async Task StatusCodePagesMiddleware.Invoke(HttpContext context)
  File "ExceptionHandlerMiddleware.cs", in async Task ExceptionHandlerMiddleware.Invoke(HttpContext context)+Awaited(?)

I’m aware this isn’t really going to help much in terms of reproduction, unfortunately there is 0 results for this error in any environment except production, so it is something load-based I am guessing… any ideas to pinpoint this down would be appreciated.

Further technical details

Npgsql version: 6.0.7 PostgreSQL version: postgres:15-bullseye docker image (although this was happening before using docker w/ postgres 12, so isn’t a docker issue) Operating system: Debian Bullseye

Example of grepping this in the logs…

2022-01-01 03:26:55.577 +00:00 [WRN] Internal error with xxx: Npgsql.NpgsqlException (0x80004005): Unknown message code: 56
2022-01-01 13:01:45.628 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 255
2022-01-01 21:34:57.002 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 4
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 2
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 0
Npgsql.NpgsqlException (0x80004005): Unknown message code: 16
Npgsql.NpgsqlException (0x80004005): Unknown message code: 54
Npgsql.NpgsqlException (0x80004005): Unknown message code: 57
Npgsql.NpgsqlException (0x80004005): Unknown message code: 54
2022-01-02 16:24:18.384 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 109
2022-01-02 16:24:22.643 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
2022-01-02 16:24:27.015 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 103
Npgsql.NpgsqlException (0x80004005): Unknown message code: 101
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 101
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
2022-01-02 16:24:30.063 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
2022-01-02 16:24:30.893 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
2022-01-02 16:24:40.774 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
2022-01-02 16:24:56.211 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 118
2022-01-02 16:24:57.401 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
2022-01-02 16:24:58.346 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 103
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 118
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
2022-01-02 16:25:14.083 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 103
2022-01-02 16:25:16.641 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 101
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
2022-01-02 16:25:17.903 +00:00 [WRN] Npgsql.NpgsqlException (0x80004005): Unknown message code: 118
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 118
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 53
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 118
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 125
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44
Npgsql.NpgsqlException (0x80004005): Unknown message code: 34
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 118
Npgsql.NpgsqlException (0x80004005): Unknown message code: 97
Npgsql.NpgsqlException (0x80004005): Unknown message code: 58
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 48
Npgsql.NpgsqlException (0x80004005): Unknown message code: 125
Npgsql.NpgsqlException (0x80004005): Unknown message code: 44