workflow-core: Deadlock Exception in EntityFramework with SQL Server provider

Since I have upgraded my solution to .net core 3.1 and to the newest workflow core libraries, I get a sporadic deadlock exception while trying to read a workflow instance.

var internalWorkflow = await workflowStore.GetWorkflowInstance(workflowId).ConfigureAwait(false); or var internalWorkflows = await workflowStore.GetWorkflowInstances(workflowIds).ConfigureAwait(false);

We are using the SQL Server persistence and lock provider.

Did I missed something in the upgrade? How can I handle this issue?

Microsoft.Data.SqlClient.SqlException
  HResult=0x80131904
  Message=Transaction (Process ID 62) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
  Source=Core Microsoft SqlClient Data Provider
  StackTrace:
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at Microsoft.Data.SqlClient.SqlDataReader.<>c__DisplayClass195_0.<ReadAsync>b__1(Task t)
   at Microsoft.Data.SqlClient.SqlDataReader.InvokeRetryable[T](Func`2 moreFunc, TaskCompletionSource`1 source, IDisposable objectToDispose)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.AsyncEnumerator.<MoveNextAsync>d__17.MoveNext() in /_/src/EFCore.Relational/Query/Internal/QueryingEnumerable.cs:line 291
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 152
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.<SingleAsync>d__21`1.MoveNext() in /_/src/EFCore/Query/ShapedQueryCompilingExpressionVisitor.cs:line 115
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.<SingleAsync>d__21`1.MoveNext() in /_/src/EFCore/Query/ShapedQueryCompilingExpressionVisitor.cs:line 127
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 152
   at WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider.<GetWorkflowInstance>d__8.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 63
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 180
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 152
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/TaskAwaiter.cs:line 563
   at WorkflowServer.Logic.Services.WorkflowService.<GetWorkflowById>d__27.MoveNext() in C:\Repos\WorkflowServer\Logic\WorkflowServer.Logic\Services\WorkflowService.cs:line 595

  This exception was originally thrown at this call stack:
	Microsoft.Data.SqlClient.SqlConnection.OnError(Microsoft.Data.SqlClient.SqlException, bool, System.Action<System.Action>)
	Microsoft.Data.SqlClient.SqlInternalConnection.OnError(Microsoft.Data.SqlClient.SqlException, bool, System.Action<System.Action>)
	Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(Microsoft.Data.SqlClient.TdsParserStateObject, bool, bool)
	Microsoft.Data.SqlClient.TdsParser.TryRun(Microsoft.Data.SqlClient.RunBehavior, Microsoft.Data.SqlClient.SqlCommand, Microsoft.Data.SqlClient.SqlDataReader, Microsoft.Data.SqlClient.BulkCopySimpleResultSet, Microsoft.Data.SqlClient.TdsParserStateObject, out bool)
	Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(out bool)
	Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(bool, out bool)
	Microsoft.Data.SqlClient.SqlDataReader.ReadAsync.AnonymousMethod__1(System.Threading.Tasks.Task)
	Microsoft.Data.SqlClient.SqlDataReader.InvokeRetryable<T>(System.Func<System.Threading.Tasks.Task, System.Threading.Tasks.Task<T>>, System.Threading.Tasks.TaskCompletionSource<T>, System.IDisposable)
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() in ExceptionDispatchInfo.cs
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) in TaskAwaiter.cs
    ...
    [Call Stack Truncated]

 	...
 	[Resuming Async Method]	
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 172	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<WorkflowServer.Common.Entities.Workflow.Workflow>.AsyncStateMachineBox<WorkflowServer.Logic.Services.WorkflowService.<GetWorkflowById>d__27>.MoveNext(System.Threading.Thread threadPoolThread) Line 617	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 304	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 742	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3326	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 2076	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetException(object exceptionObject) Line 3226	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<WorkflowCore.Models.WorkflowInstance>.SetException(System.Exception exception) Line 784	C#
 	WorkflowCore.Persistence.EntityFramework.dll!WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider.GetWorkflowInstance(string Id)	Unknown
 	WorkflowCore.Persistence.EntityFramework.dll!WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider.GetWorkflowInstance(string Id)	Unknown
 	[Resuming Async Method]	
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 172	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<WorkflowCore.Models.WorkflowInstance>.AsyncStateMachineBox<WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider.<GetWorkflowInstance>d__8>.MoveNext(System.Threading.Thread threadPoolThread) Line 617	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 304	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 742	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3326	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 2076	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetException(object exceptionObject) Line 3226	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>.SetException(System.Exception exception) Line 784	C#
 	Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>(System.Collections.Generic.IAsyncEnumerable<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow> asyncEnumerable, System.Threading.CancellationToken cancellationToken) Line 127	C#
 	Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleAsync<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>(System.Collections.Generic.IAsyncEnumerable<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow> asyncEnumerable, System.Threading.CancellationToken cancellationToken) Line 127	C#
 	[Resuming Async Method]	
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 172	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>.AsyncStateMachineBox<Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.<SingleAsync>d__21<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>>.MoveNext(System.Threading.Thread threadPoolThread) Line 617	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 304	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 742	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3326	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 2076	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetException(object exceptionObject) Line 3226	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.SetException(System.Exception exception) Line 784	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<bool>.SetException(System.Exception exception) Line 164	C#
 	Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>.AsyncEnumerator.MoveNextAsync() Line 289	C#
 	Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>.AsyncEnumerator.MoveNextAsync() Line 289	C#
 	[Resuming Async Method]	
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 172	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable<WorkflowCore.Persistence.EntityFramework.Models.PersistedWorkflow>.AsyncEnumerator.<MoveNextAsync>d__17>.MoveNext(System.Threading.Thread threadPoolThread) Line 617	C#
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 304	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 742	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 3326	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<bool>.TrySetResult(bool result) Line 419	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.TaskCompletionSource<bool>.TrySetResult(bool result) Line 271	C#
 	Microsoft.Data.SqlClient.dll!Microsoft.Data.SqlClient.SqlDataReader.CompleteRetryable<bool>(System.Threading.Tasks.Task<bool> task, System.Threading.Tasks.TaskCompletionSource<bool> source, System.IDisposable objectToDispose)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 289	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread) Line 2389	C#
 	System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 663	C#
 	[Async Call Stack]	
 	...

About this issue

Original URL
State: open
Created 4 years ago
Comments: 17 (7 by maintainers)

Most upvoted comments

I also couldn’t reproduce it with the latest patch. But I noticed some exceptions in the Application Insights logs of our running instance on Azure. I will have a look if they still occur with your patch.

Would it be possible to extend your extension UseSqlServer(...) with the possibility to set EnableRetryOnFailure() as it is proposed in the exception message?

Microsoft.Data.SqlClient.SqlException at WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider+<GetRunnableInstances>d__6.MoveNext

An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure()' to the 'UseSqlServer' call. Database 'sql-DB001' on server 'sql-srv' is not currently available.  Please retry the connection later. 

System.InvalidOperationException:
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext (Microsoft.EntityFrameworkCore.SqlServer, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+AsyncEnumerator+<MoveNextAsync>d__17.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__64`1.MoveNext (Microsoft.EntityFrameworkCore, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__64`1.MoveNext (Microsoft.EntityFrameworkCore, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider+<GetRunnableInstances>d__6.MoveNext (WorkflowCore.Persistence.EntityFramework, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at WorkflowCore.Services.BackgroundTasks.RunnablePoller+<PollRunnables>d__9.MoveNext (WorkflowCore, Version=3.1.5.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at WorkflowCore.Services.BackgroundTasks.RunnablePoller+<PollRunnables>d__9.MoveNext (WorkflowCore, Version=3.1.5.0, Culture=neutral, PublicKeyToken=null)
Inner exception Microsoft.Data.SqlClient.SqlException handled at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext:
   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlConnection.OpenAsync (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenDbConnectionAsync>d__50.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenDbConnectionAsync>d__50.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenAsync>d__47.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+AsyncEnumerator+<InitializeReaderAsync>d__18.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext (Microsoft.EntityFrameworkCore.SqlServer, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

Microsoft.Data.SqlClient.SqlException at WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider+<GetRunnableEvents>d__16.MoveNext

An exception has been raised that is likely due to a transient failure. Consider enabling transient error resiliency by adding 'EnableRetryOnFailure()' to the 'UseSqlServer' call. Database 'sql-DB001' on server 'sql-srv' is not currently available.  Please retry the connection later. 

System.InvalidOperationException:
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext (Microsoft.EntityFrameworkCore.SqlServer, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+AsyncEnumerator+<MoveNextAsync>d__17.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__64`1.MoveNext (Microsoft.EntityFrameworkCore, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions+<ToListAsync>d__64`1.MoveNext (Microsoft.EntityFrameworkCore, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at WorkflowCore.Persistence.EntityFramework.Services.EntityFrameworkPersistenceProvider+<GetRunnableEvents>d__16.MoveNext (WorkflowCore.Persistence.EntityFramework, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at WorkflowCore.Services.BackgroundTasks.RunnablePoller+<PollRunnables>d__9.MoveNext (WorkflowCore, Version=3.1.5.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at WorkflowCore.Services.BackgroundTasks.RunnablePoller+<PollRunnables>d__9.MoveNext (WorkflowCore, Version=3.1.5.0, Culture=neutral, PublicKeyToken=null)
Inner exception Microsoft.Data.SqlClient.SqlException handled at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext:
   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen (Microsoft.Data.SqlClient, Version=1.0.19269.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenDbConnectionAsync>d__50.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenDbConnectionAsync>d__50.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection+<OpenAsync>d__47.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand+<ExecuteReaderAsync>d__17.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1+AsyncEnumerator+<InitializeReaderAsync>d__18.MoveNext (Microsoft.EntityFrameworkCore.Relational, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy+<ExecuteAsync>d__7`2.MoveNext (Microsoft.EntityFrameworkCore.SqlServer, Version=3.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

knoxi on Feb 26, 2020

Hello, I have the exact same error after upgrading to 3.1.

Regards, Christoph

xophp on Feb 10, 2020