Umbraco-CMS: Slow performance and deadlocks in SignInAsync during high load (v.10–13–?)

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

10.1

Bug summary

We experiencing deadlocks in the database when lots of members sign in at the same time. During peaks we have around 100 sign ins a minute. We use umbracos default membership provider. The sign ins get very slow and we also see deadlocks in the logs.

Specifics

Setting the logger in debug mode locally I got the following two stacktraces.

Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 81) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainWriteLock()
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
   at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.WriteLock(Int32 lockId, Nullable`1 obtainLockTimeout)
   at Umbraco.Cms.Infrastructure.Scoping.Scope.ObtainWriteLock(Int32 lockId, Nullable`1 timeout)
   at Umbraco.Cms.Infrastructure.Scoping.Scope.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Nullable`1 timeout, Int32 lockId)
   at Umbraco.Cms.Infrastructure.Scoping.Scope.EagerWriteLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)
   at Umbraco.Cms.Infrastructure.Scoping.Scope.EnsureDbLocks()
   at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
   at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
   at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.MemberRepository.PersistUpdatedItem(IMember entity)
   at Umbraco.Cms.Core.Cache.DefaultRepositoryCachePolicy`2.Update(TEntity entity, Action`1 persistUpdated)
   at Umbraco.Cms.Core.Services.MemberService.Save(IMember member)
   at Umbraco.Cms.Core.Security.MemberUserStore.UpdateAsync(MemberIdentityUser user, CancellationToken cancellationToken)
ClientConnectionId:27b072f9-dd7a-40fb-8c14-93950869d88c
Error Number:1205,State:51,Class:13
at System.Environment.get_StackTrace()at 
Umbraco.Cms.Infrastructure.Persistence.UmbracoDatabase.OnException(Exception ex)at 
NPoco.Database.OnExceptionInternal(Exception exception)at 
NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)at 
Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainWriteLock()at 
Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)at 
Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.WriteLock(Int32 lockId, Nullable`1 obtainLockTimeout)at 
Umbraco.Cms.Infrastructure.Scoping.Scope.ObtainWriteLock(Int32 lockId, Nullable`1 timeout)at 
Umbraco.Cms.Infrastructure.Scoping.Scope.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Nullable`1 timeout, Int32 lockId)at 
Umbraco.Cms.Infrastructure.Scoping.Scope.EagerWriteLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)at 
Umbraco.Cms.Infrastructure.Scoping.Scope.EnsureDbLocks()at 
Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()at 
Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()at 
Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.MemberRepository.PersistUpdatedItem(IMember entity)at 
Umbraco.Cms.Core.Cache.DefaultRepositoryCachePolicy`2.Update(TEntity entity, Action`1 persistUpdated)at 
Umbraco.Cms.Core.Services.MemberService.Save(IMember member)at 
Umbraco.Cms.Core.Security.MemberUserStore.UpdateAsync(MemberIdentityUser user, CancellationToken cancellationToken)at 
Microsoft.AspNetCore.Identity.UserManager`1.UpdateUserAsync(TUser user)at 
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)at 
Microsoft.AspNetCore.Identity.UserManager`1.UpdateUserAsync(TUser user)at 
Microsoft.AspNetCore.Identity.UserManager`1.UpdateAsync(TUser user)at 
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.HandleSignIn(TUser user, String username, SignInResult result)at 
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)at 
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.HandleSignIn(TUser user, String username, SignInResult result)at 
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.SignInWithClaimsAsync(TUser user, AuthenticationProperties authenticationProperties, IEnumerable`1 additionalClaims)at 
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)at 
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.SignInWithClaimsAsync(TUser user, AuthenticationProperties authenticationProperties, IEnumerable`1 additionalClaims)at 
Microsoft.AspNetCore.Identity.SignInManager`1.SignInAsync(TUser user, AuthenticationProperties authenticationProperties, String authenticationMethod)at 
Microsoft.AspNetCore.Identity.SignInManager`1.SignInAsync(TUser user, Boolean isPersistent, String authenticationMethod)at 
XXX.Web.Business.Helpers.AuthenticationHelper.SignInToUmbracoAsync(LoginResult clientLogin) in AuthenticationHelper.cs:line 232at 

Steps to reproduce

I can recreate it locally by using a load testing framework (we used https://locust.io/).

Expected result / actual result

No response


This item has been added to our backlog AB#52361

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 6
  • Comments: 19 (13 by maintainers)

Most upvoted comments

Resolved via https://github.com/umbraco/Umbraco-CMS/pull/19308 and targeted for 13.9 and 16.1.