aspnetcore: AddDataProtection race condition

I’m running multiple instances of a .NET Core 2.1 app behind a load balancer.

When multiple instances are created for the first time, there are frequently race conditions taking place around the DataProtection library that cause problems.

If these instances are created at the same time, the first one will create a new key and persist it to Redis, but the second one will do the same, and the third one will do the same. What you end up with is multiple keys in the key ring that are not known by many of the instances. If each instance is spinned up after a couple of seconds, then this problem doesn’t occur.

If you see the logs below, there are three default keys used by different instances.

Given that for some reason the keys are created with an immediate activation date, in many cases, they don’t get propagated to all the instances. The problem you’ll end up facing is that if you encrypt a cookie with a key that is not available on another instance, you won’t be able to decrypt it if you get to one of those instances, getting exceptions like this:

[15:29:57 ERR] An unhandled exception has occurred while executing the request.
System.Security.Cryptography.CryptographicException: The key {8480fbc2-3e02-4b31-b137-dfc300311e24} was not found in the key ring.
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.SetCobrand(ActionExecutingContext context) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 64
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 40
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[15:29:57 ERR] An exception was thrown attempting to execute the error handler.
System.Security.Cryptography.CryptographicException: The key {8480fbc2-3e02-4b31-b137-dfc300311e24} was not found in the key ring.
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.SetCobrand(ActionExecutingContext context) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 64
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 40
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[15:29:57 ERR] Connection id "0HLGK1LMKD07L", Request id "0HLGK1LMKD07L:00000003": An unhandled exception was thrown by the application.
System.Security.Cryptography.CryptographicException: The key {8480fbc2-3e02-4b31-b137-dfc300311e24} was not found in the key ring.
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.SetCobrand(ActionExecutingContext context) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 64
at MyApp.Worker.Authentication.WebApp.Filters.ClientSettingsAsyncFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) in /src/src/MyApp.Worker.Authentication.WebApp/Filters/ClientSettingsAsyncFilter.cs:line 40
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at IdentityServer4.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IEndpointRouter router, IUserSession session, IEventService events)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

I could come up with dirty hacks to avoid this, either at a code level or infrastructure level, but I’d like to know what the recommended approach is and how this can be properly avoided going forward.

• Microsoft.NETCore.App version: 2.1.0
• dotnet -info:

.NET Core SDK (reflecting any global.json):
 Version:   2.1.302
 Commit:    9048955601

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.302\

Host (useful for support):
  Version: 2.1.2
  Commit:  811c3ce6c0

.NET Core SDKs installed:
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Here are the relevant parts of my Startup.cs file:

var signingCertificate = new X509Certificate2(identityConfiguration.Credentials.FilePath, identityConfiguration.Credentials.Password, X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.EphemeralKeySet);

var redis = ConnectionMultiplexer.Connect(Configuration.GetConnectionString("Redis"));

services
    .AddDataProtection()
    .SetApplicationName(ApplicationName)
    .PersistKeysToRedis(redis, $"{ApplicationName}-DataProtectionKeys")
    .ProtectKeysWithCertificate(signingCertificate);

And here are the relevant parts of the logs of 8 instances when they are started:

Instance 1:

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z.
[15:28:37 DBG] Descriptor deserializer type for key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Considering key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} with expiration date 2018-12-05 15:28:36Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} as the default key.
[15:28:37 DBG] Key ring with default key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} was loaded during application startup.

Instance 2:

[15:28:36 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:36 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:36 INF] Creating key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef} with creation date 2018-09-06 15:28:36Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z.
[15:28:37 DBG] Descriptor deserializer type for key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Considering key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} with expiration date 2018-12-05 15:28:36Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} as the default key.
[15:28:37 DBG] Key ring with default key {6ebc8595-fcc1-408a-8a44-14c4fac21df0} was loaded during application startup.

Instance 3:

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {fdb8d6da-d846-429e-a77a-612320851ff4} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z.
[15:28:37 DBG] Descriptor deserializer type for key {fdb8d6da-d846-429e-a77a-612320851ff4} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {fdb8d6da-d846-429e-a77a-612320851ff4} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}.
[15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}.
[15:28:37 DBG] Considering key {4ab699bc-a747-4452-8b4e-1df6828f7b30} with expiration date 2018-12-05 15:28:37Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {4ab699bc-a747-4452-8b4e-1df6828f7b30} as the default key.
[15:28:37 DBG] Key ring with default key {4ab699bc-a747-4452-8b4e-1df6828f7b30} was loaded during application startup.

Instance 4:

[15:28:36 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:36Z, and expiration date 2018-12-05 15:28:36Z.
[15:28:37 DBG] Descriptor deserializer type for key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}.
[15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}.
[15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}.
[15:28:37 DBG] Considering key {4ab699bc-a747-4452-8b4e-1df6828f7b30} with expiration date 2018-12-05 15:28:37Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {4ab699bc-a747-4452-8b4e-1df6828f7b30} as the default key.
[15:28:37 DBG] Key ring with default key {4ab699bc-a747-4452-8b4e-1df6828f7b30} was loaded during application startup.

Instance 5:

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z.
[15:28:37 DBG] Descriptor deserializer type for key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}.
[15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}.
[15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}.
[15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}.
[15:28:37 DBG] Considering key {4ab699bc-a747-4452-8b4e-1df6828f7b30} with expiration date 2018-12-05 15:28:37Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {4ab699bc-a747-4452-8b4e-1df6828f7b30} as the default key.
[15:28:37 DBG] Key ring with default key {4ab699bc-a747-4452-8b4e-1df6828f7b30} was loaded during application startup.

Instance 6:

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {8480fbc2-3e02-4b31-b137-dfc300311e24} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z.
[15:28:37 DBG] Descriptor deserializer type for key {8480fbc2-3e02-4b31-b137-dfc300311e24} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {8480fbc2-3e02-4b31-b137-dfc300311e24} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}.
[15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}.
[15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}.
[15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}.
[15:28:37 DBG] Found key {8480fbc2-3e02-4b31-b137-dfc300311e24}.
[15:28:37 DBG] Considering key {8480fbc2-3e02-4b31-b137-dfc300311e24} with expiration date 2018-12-05 15:28:37Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {8480fbc2-3e02-4b31-b137-dfc300311e24} as the default key.
[15:28:37 DBG] Key ring with default key {8480fbc2-3e02-4b31-b137-dfc300311e24} was loaded during application startup.

Instance 7:

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {c845406b-d630-43a3-949b-e0387ddfd377} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z.
[15:28:37 DBG] Descriptor deserializer type for key {c845406b-d630-43a3-949b-e0387ddfd377} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {c845406b-d630-43a3-949b-e0387ddfd377} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}.
[15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}.
[15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}.
[15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}.
[15:28:37 DBG] Found key {8480fbc2-3e02-4b31-b137-dfc300311e24}.
[15:28:37 DBG] Found key {4f871aa5-0440-4e21-b21f-a40878f2f936}.
[15:28:37 DBG] Found key {c845406b-d630-43a3-949b-e0387ddfd377}.
[15:28:37 DBG] Considering key {8480fbc2-3e02-4b31-b137-dfc300311e24} with expiration date 2018-12-05 15:28:37Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {8480fbc2-3e02-4b31-b137-dfc300311e24} as the default key.
[15:28:37 DBG] Key ring with default key {8480fbc2-3e02-4b31-b137-dfc300311e24} was loaded during application startup.

Instance 8:

[15:28:37 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:28:37 DBG] Policy resolution states that a new key should be added to the key ring.
[15:28:37 INF] Creating key {17bd9124-6a55-43d3-97ba-ec2543667a7e} with creation date 2018-09-06 15:28:37Z, activation date 2018-09-06 15:28:37Z, and expiration date 2018-12-05 15:28:37Z.
[15:28:37 DBG] Descriptor deserializer type for key {17bd9124-6a55-43d3-97ba-ec2543667a7e} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:28:37 DBG] No key escrow sink found. Not writing key {17bd9124-6a55-43d3-97ba-ec2543667a7e} to escrow.
[15:28:37 DBG] Encrypting to X.509 certificate with thumbprint 'XXXXX'.
[15:28:37 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:28:37 DBG] Found key {6ebc8595-fcc1-408a-8a44-14c4fac21df0}.
[15:28:37 DBG] Found key {b5af49fe-11d5-4fa7-aa61-85fb5f4d1bef}.
[15:28:37 DBG] Found key {4ab699bc-a747-4452-8b4e-1df6828f7b30}.
[15:28:37 DBG] Found key {fdb8d6da-d846-429e-a77a-612320851ff4}.
[15:28:37 DBG] Found key {e5bd64e5-d825-4fb1-8517-a5b9eb0da5a0}.
[15:28:37 DBG] Found key {dac23a76-9aaa-4dae-be25-ecfcc2ea28a9}.
[15:28:37 DBG] Found key {8480fbc2-3e02-4b31-b137-dfc300311e24}.
[15:28:37 DBG] Found key {4f871aa5-0440-4e21-b21f-a40878f2f936}.
[15:28:37 DBG] Found key {c845406b-d630-43a3-949b-e0387ddfd377}.
[15:28:37 DBG] Found key {17bd9124-6a55-43d3-97ba-ec2543667a7e}.
[15:28:37 DBG] Considering key {8480fbc2-3e02-4b31-b137-dfc300311e24} with expiration date 2018-12-05 15:28:37Z as default key.
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Forwarded activator type request from Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 to Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Culture=neutral, PublicKeyToken=adb9793829ddae60
[15:28:37 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:28:37 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:28:37 DBG] Using key {8480fbc2-3e02-4b31-b137-dfc300311e24} as the default key.
[15:28:37 DBG] Key ring with default key {8480fbc2-3e02-4b31-b137-dfc300311e24} was loaded during application startup.