Umbraco-CMS: Slow performance and deadlocks in SignInAsync during high load (v.10–13–?)
Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)
10.1
Bug summary
We experiencing deadlocks in the database when lots of members sign in at the same time. During peaks we have around 100 sign ins a minute. We use umbracos default membership provider. The sign ins get very slow and we also see deadlocks in the logs.
Specifics
Setting the logger in debug mode locally I got the following two stacktraces.
Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 81) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at Umbraco.Cms.Infrastructure.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainWriteLock()
at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)
at Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.WriteLock(Int32 lockId, Nullable`1 obtainLockTimeout)
at Umbraco.Cms.Infrastructure.Scoping.Scope.ObtainWriteLock(Int32 lockId, Nullable`1 timeout)
at Umbraco.Cms.Infrastructure.Scoping.Scope.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Nullable`1 timeout, Int32 lockId)
at Umbraco.Cms.Infrastructure.Scoping.Scope.EagerWriteLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)
at Umbraco.Cms.Infrastructure.Scoping.Scope.EnsureDbLocks()
at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
at Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()
at Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.MemberRepository.PersistUpdatedItem(IMember entity)
at Umbraco.Cms.Core.Cache.DefaultRepositoryCachePolicy`2.Update(TEntity entity, Action`1 persistUpdated)
at Umbraco.Cms.Core.Services.MemberService.Save(IMember member)
at Umbraco.Cms.Core.Security.MemberUserStore.UpdateAsync(MemberIdentityUser user, CancellationToken cancellationToken)
ClientConnectionId:27b072f9-dd7a-40fb-8c14-93950869d88c
Error Number:1205,State:51,Class:13
at System.Environment.get_StackTrace()at
Umbraco.Cms.Infrastructure.Persistence.UmbracoDatabase.OnException(Exception ex)at
NPoco.Database.OnExceptionInternal(Exception exception)at
NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)at
Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock.ObtainWriteLock()at
Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.SqlServerDistributedLock..ctor(SqlServerDistributedLockingMechanism parent, Int32 lockId, DistributedLockType lockType, TimeSpan timeout)at
Umbraco.Cms.Persistence.SqlServer.Services.SqlServerDistributedLockingMechanism.WriteLock(Int32 lockId, Nullable`1 obtainLockTimeout)at
Umbraco.Cms.Infrastructure.Scoping.Scope.ObtainWriteLock(Int32 lockId, Nullable`1 timeout)at
Umbraco.Cms.Infrastructure.Scoping.Scope.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`2 obtainLock, Nullable`1 timeout, Int32 lockId)at
Umbraco.Cms.Infrastructure.Scoping.Scope.EagerWriteLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)at
Umbraco.Cms.Infrastructure.Scoping.Scope.EnsureDbLocks()at
Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()at
Umbraco.Cms.Infrastructure.Scoping.Scope.get_Database()at
Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.MemberRepository.PersistUpdatedItem(IMember entity)at
Umbraco.Cms.Core.Cache.DefaultRepositoryCachePolicy`2.Update(TEntity entity, Action`1 persistUpdated)at
Umbraco.Cms.Core.Services.MemberService.Save(IMember member)at
Umbraco.Cms.Core.Security.MemberUserStore.UpdateAsync(MemberIdentityUser user, CancellationToken cancellationToken)at
Microsoft.AspNetCore.Identity.UserManager`1.UpdateUserAsync(TUser user)at
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)at
Microsoft.AspNetCore.Identity.UserManager`1.UpdateUserAsync(TUser user)at
Microsoft.AspNetCore.Identity.UserManager`1.UpdateAsync(TUser user)at
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.HandleSignIn(TUser user, String username, SignInResult result)at
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)at
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.HandleSignIn(TUser user, String username, SignInResult result)at
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.SignInWithClaimsAsync(TUser user, AuthenticationProperties authenticationProperties, IEnumerable`1 additionalClaims)at
System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)at
Umbraco.Cms.Web.Common.Security.UmbracoSignInManager`1.SignInWithClaimsAsync(TUser user, AuthenticationProperties authenticationProperties, IEnumerable`1 additionalClaims)at
Microsoft.AspNetCore.Identity.SignInManager`1.SignInAsync(TUser user, AuthenticationProperties authenticationProperties, String authenticationMethod)at
Microsoft.AspNetCore.Identity.SignInManager`1.SignInAsync(TUser user, Boolean isPersistent, String authenticationMethod)at
XXX.Web.Business.Helpers.AuthenticationHelper.SignInToUmbracoAsync(LoginResult clientLogin) in AuthenticationHelper.cs:line 232at
Steps to reproduce
I can recreate it locally by using a load testing framework (we used https://locust.io/).
Expected result / actual result
No response
This item has been added to our backlog AB#52361
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 6
- Comments: 19 (13 by maintainers)
Resolved via https://github.com/umbraco/Umbraco-CMS/pull/19308 and targeted for 13.9 and 16.1.