aspnetcore: TaskCanceled Exception thrown from Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler class

Dear all,

We have a .NET Core Web API service which makes use of the Authentication middleware readily available with .NET 5.0. In our service we have registered 4 different jwt bearer authentication schemes on startup of the service, (why? - that’s because we are in the process of migrating from two older identity providers to two newer ones. For example, we are migrating from Azure AD V1 auth endpoints to the Microsoft Identity Platform (V2)).

In a number of our deployments we see TaskCanceled exceptions with the below stack trace, when certain api’s are called. The api’s are all protected by an [Authorize] attribute which makes use of the aforementioned auth schemes that are registered at startup

Details of the service: SDK version: .NET 5.0.401 Base docker image: mcr.microsoft.com/dotnet/aspnet:5.0.10

We always seem to have 2 of these exceptions thrown simultaeneously:

System.Threading.Tasks.TaskCanceledException:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1+<GetConfigurationAsync>d__24.MoveNext (Microsoft.IdentityModel.Protocols, Version=6.7.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=5.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=5.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1+<AuthenticateAsync>d__48.MoveNext (Microsoft.AspNetCore.Authentication, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationService+<AuthenticateAsync>d__14.MoveNext (Microsoft.AspNetCore.Authentication.Core, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.Policy.PolicyEvaluator+<AuthenticateAsync>d__2.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Startup+<>c+<<Configure>b__9_0>d.MoveNext

And the second exception stack trace:

  System.Threading.Tasks.TaskCanceledException:
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1+<GetConfigurationAsync>d__24.MoveNext (Microsoft.IdentityModel.Protocols, Version=6.7.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=5.0.10.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

The troublesome thing is that I can’t seem to reproduce this error intentionally. We set up suitable load tests to test the affected api’s, but we don’t seem to be able to reproduce the failures as part of the load tests, but the errors keep happening in the same environments. As far as I can tell we have configured the middleware correctly as for the majority of our calls we are able to authenticate and authorize correctly (we have automated tests testing various scenarios).

But in certain sporadic cases, we get these exceptions.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 29 (12 by maintainers)

Most upvoted comments

Can you open an issue in https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/?

+1
Tratcher on Feb 23, 2023

RequestRefresh is supposed to update _syncAfter. Any request after that gets queued waiting for new config. That’s by design, though there’s some plans to allow requests to continue with the last known good config.

One place I see that could be improved is here: https://github.com/AzureAD/azure-activedirectory-identitymodel-extensions-for-dotnet/blob/9e6f90a9a8032107af6f5c851e9d257f4c6c8fdc/src/Microsoft.IdentityModel.Protocols/Configuration/ConfigurationManager.cs#L136-L143 Right now it assumes the _refreshLock is acquired quickly and now still current. However, if the refresh takes too long and/or there’s lock contention, then now could be outdated. `now should be updated inside the lock (line 136), and again after the refresh finishes (line 143). Once the refresh completes that should help all other requests waiting on the lock to quickly exit.

The only other thing that could be changed is to protect RequestRefresh with _refreshLock, but that risks blocking threads that don’t really need to wait for the refresh to happen.

+1
Tratcher on Feb 23, 2023

You can make the API call with a client like HttpClient or Curl and then cancel it.

+1
Tratcher on Nov 18, 2021
Read more comments on GitHub
← aspnetcore: System.ObjectDisposedException: Cannot write to the response body, the response has completed
aspnetcore: Tasks returned from JSRuntime.InvokeAsync during OnInitAsync never finishes in preview3 →