SqlClient: "Login failed for user ***. Token is expired." when using Azure Active Directory password authentication

Describe the bug

We use Azure Active Directory password authentication and get the following exception 2-4 times per day:

Exception message:
Login failed for user '***@***.com'. Token is expired.
A severe error occurred on the current command.  The results, if any, should be discarded.

Stack trace may different. Here are two examples:
Stack trace #1
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlConnection.OnError","level":0,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning","level":1,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.TdsParser.TryRun","level":2,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData","level":3,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlDataReader.get_MetaData","level":4,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader","level":5,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds","level":6,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader","level":7,"line":0}
{"assembly":"Microsoft.Data.SqlClient, Version=2.0.20168.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlCommand.ExecuteReader","level":8,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader","level":9,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+Enumerator.InitializeReader","level":10,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.SqlServer, Version=3.1.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.Execute","level":11,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+Enumerator.MoveNext","level":12,"line":0}
{"assembly":"System.Linq, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","method":"System.Linq.Enumerable.Single","level":13,"line":0}
...

Stack trace #2 
{"assembly":"Microsoft.Data.SqlClient, Version=1.10.19324.4, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5","method":"Microsoft.Data.SqlClient.SqlCommand+<>c.<ExecuteDbDataReaderAsync>b__164_0","level":0,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke","level":1,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.ExecutionContext.RunInternal","level":2,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":3,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.Tasks.Task.ExecuteWithThreadLocal","level":4,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":5,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":6,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17.MoveNext","level":7,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":8,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17.MoveNext","level":9,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":10,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17.MoveNext","level":11,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":12,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":13,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+AsyncEnumerator+<InitializeReaderAsync>d__18.MoveNext","level":14,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":15,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":16,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.SqlServer, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext","level":17,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":18,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":19,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore.Relational, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+AsyncEnumerator+<MoveNextAsync>d__17.MoveNext","level":20,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":21,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":22,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor+<SingleOrDefaultAsync>d__22`1.MoveNext","level":23,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":24,"line":0}
{"assembly":"Microsoft.EntityFrameworkCore, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","method":"Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor+<SingleOrDefaultAsync>d__22`1.MoveNext","level":25,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":26,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":27,"line":0}
...

To reproduce

Unfortunately, I cannot reproduce the issue. I just see the exception in our logs. I will update the ticket if I manage to reproduce the bug.

Expected behavior

I don’t know, how auth token expiration is handled inside Microsoft.Data.SqlClient. But I would expect that the token is refreshed prior to expiration. Or that the expiration exception is properly handled inside the library without throwing it outside.

Further technical details

Microsoft.Data.SqlClient version: 2.0.20168.4. The exception also happened when we used the version 1.10.19324.4.

.NET target: Core 3.1.5

SQL Server version: SQL Server 2014 (12.0.2000.8) on Azure.

Operating system: Docker container based on the image mcr.microsoft.com/dotnet/core/aspnet:3.1