aspnetcore: SNAT Port Exhaustion, Microsoft.AspNetCore.Identity.UserManager Async Calls Are blocking

Describe the bug

Some methods advertised as being async are in fact not.

Our application is using MSI authentication, and uses a db connection interceptor to set the access token on connection open. when we first enabled MSI authentication or application fell over at peak due to SNAT port exhaustion.

We eventually found the cause was due to workers being blocked on opening a db connection in a synchronous call even though the expectation was it would be async. The main culprit for our case was UserManager.FindByEmailAsync

Methods used by our application that exhibit this behaviour include

1. CreateAsync
2. FindByEmailAsync
3. UpdateAsync
4. ResetPasswordAsync
5. RemoveClaimsAsync
6. AddToRoleAsync

If you set RequireUniqueEmail to off then some if not all of the above do not use the blocking call.

To Reproduce

https://github.com/sbkaspersen/jubilant-sniffle

Further technical details

• ASP.NET Core version 3.1.3
• Include the output of dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.1.302
 Commit:    41faccf259

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.302\

Host (useful for support):
  Version: 3.1.6
  Commit:  3acd9b0cd1

.NET Core SDKs installed:
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.302 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• VS 2019 Enterprise