aspnetcore: AspNetCore.Correlation.Google cookie not found

After upgrading from 3.0 to 3.1 I have issues with the Google authentication. I have no cookie policy registered and this is how my authentication configuration looks like:

services
          .AddAuthentication(options =>
          {
            options.DefaultScheme = IdentityConstants.ApplicationScheme;
            options.DefaultSignInScheme = IdentityConstants.ExternalScheme;
          })
          .AddGoogle(options =>
          {
            options.ClientId = _configurationOptions.Google.ClientId;
            options.ClientSecret = _configurationOptions.Google.ClientSecret;
            options.Scope.Add("https://www.googleapis.com/auth/userinfo.profile");
            options.SaveTokens = true;
          })
        .AddIdentityCookies(options =>
        {
          options.ApplicationCookie.Configure(a =>
          {
            a.Cookie.Name = ".AspNet.SharedCookie";
            a.Cookie.Domain = _configurationOptions.Domain;
          });
        });

And this is the error I get:

warn: Microsoft.AspNetCore.Authentication.Google.GoogleHandler[15]
      '.AspNetCore.Correlation.Google.611LD72jlzagvpKikBe8B41IqHTqrvVnXZdWS9iuZd8' cookie not found.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HLRR2LP0DVR7", Request id "0HLRR2LP0DVR7:00000001": An unhandled exception was thrown by the application.
System.Exception: An error was encountered while handling the remote login.
 ---> System.Exception: Correlation failed.
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler`1.HandleRequestAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)```