aspnetcore: DefaultHttpResponse throws null ref when trying to access Cookies, even when using null check, only from bots

Describe the bug

I’m attempting to set a cookie with the following code (source is here):

var context = _httpContextAccessor.HttpContext;
context.Response?.Cookies?.Append(CookieKey, topic.TopicID.ToString());

…where the context accessor is inserted via DI. However, per the stack trace listed below, Microsoft.AspNetCore.Http.DefaultHttpResponse.get_Cookies() throws a null ref exception. What’s even more strange is that it only occurs with search indexing bots. So far I’ve logged errors from Bing, Google, webmeup, brandwatch, mj12bot, semrush, grapeshot and others. It’s always a bot.

My assumption is that even if the Cookies property in my code was null, it shouldn’t fail, and stack trace is throwing from within the DefaultHttpResponse code. Two interesting points though:

It’s always a bot when this error is logged.
It seems to have started when I migrated the site to a Linux app service on Azure (was previously a Windows based app service on Azure).

To Reproduce

I can’t reproduce this locally. While I’m recording the full headers of the search bots, nothing stands out as a reason that for the cookie get on DefaultHttpResponse to throw. For fun though, here are some typical headers, this one from Bing:

Cache-Control: no-cache
Connection: Keep-Alive
Pragma: no-cache
Accept: */*
Accept-Encoding: gzip, deflate
From: bingbot(at)microsoft.com
Host: coasterbuzz.com
Max-Forwards: 10
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
X-Client-IP: 40.77.167.62
X-Client-Port: 17849
X-WAWS-Unencoded-URL: /Forums/Topic/first-park-for-the-2002-season
CLIENT-IP: 40.77.167.62:17849
X-ARR-LOG-ID: 66fbfb17-df14-425f-8571-192af0f32d6d
DISGUISED-HOST: coasterbuzz.com
X-SITE-DEPLOYMENT-ID: coasterbuzz
WAS-DEFAULT-HOSTNAME: coasterbuzz.azurewebsites.net
X-Original-URL: /Forums/Topic/first-park-for-the-2002-season
X-Original-Proto: http
X-ARR-SSL: 4096|256|C=GB, S=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA|CN=coasterbuzz.com
X-AppService-Proto: https
X-Forwarded-TlsVersion: 1.2
X-Original-For: [::ffff:172.16.3.1]:11898

Exceptions (if any)

NullReferenceException: Object reference not set to an instance of an object.

   at Microsoft.AspNetCore.Http.DefaultHttpResponse.get_Cookies()
   at PopForums.Mvc.Areas.Forums.Services.TopicViewCountService.SetViewedTopic(Topic topic)
   at PopForums.Mvc.Areas.Forums.Services.TopicViewCountService.ProcessView(Topic topic)
   at PopForums.Mvc.Areas.Forums.Controllers.ForumController.Topic(String id, Int32 pageNumber)
   at lambda_method(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

Further technical details

ASP.NET Core version: 3.1.7 (SDK 3.1.401)
Running on Linux Azure App Service, self-contained deployment

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 39 (23 by maintainers)

Most upvoted comments

I think I’m seeing a similar bug, or at least a similar effect.

We’re running integration tests on the dotnet core 3.1-buster docker image (which looks like it’s on dotnet sdk 3.1.403).

We have some tests checking authorisation logic which occasionally fail with a null reference on the last line of the below snippet:

var context = ContextAccessor.HttpContext;
var request = context.Request;
var query = request.Query;

The stack trace shows that it hits DefaultHttpRequest and fails on the Query get, and digging through the code I independently came to the same conclusion as @benaadams above - that something somewhere is calling Uninitialize on the request. Googling that issue was what brought me here.

What also stood out is the comment from @jeffputz that it was only when flipping to Linux - we only see it on our CI server (running a Linux docker image), and never running tests locally on windows machines, so that would explain that. I’ll try doing some runs via the same docker image locally to see if I can reproduce it that way.

I appreciate there’s not much to go on so far, and the root causes may be different. Just giving you another data point in case it helps at all. I’ll let you know if I find anything else too, and let me know if any more info would be helpful.

mattcottam91 on Nov 6, 2020