microsoft-identity-web: UpdateMergedOptionsFromMicrosoftIdentityOptions Collection was modified; enumeration operation may not execute.

@jennyf19 @jmprieur Unfortunately, the issue was not fixed. The problem might be reproduced on the same test using v1.25.8, just run it a couple of times. Stacktrace:

System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value, Int32& location)
   at System.Collections.Generic.HashSet`1.System.Collections.Generic.ICollection<T>.Add(T item)
   at Microsoft.Identity.Web.MergedOptions.UpdateMergedOptionsFromMicrosoftIdentityOptions(MicrosoftIdentityOptions microsoftIdentityOptions, MergedOptions mergedOptions)
   at Microsoft.Identity.Web.MicrosoftIdentityWebApiAuthenticationBuilderExtensions.<>c__DisplayClass3_0.<AddMicrosoftIdentityWebApiImplementation>b__0(JwtBearerOptions options, IServiceProvider serviceProvider, IOptionsMonitor`1 mergedOptionsMonitor, IOptionsMonitor`1 msIdOptionsMonitor, IOptions`1 msIdOptions)
   at Microsoft.Extensions.Options.ConfigureNamedOptions`5.Configure(String name, TOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsCache`1.<>c__3`1.<GetOrAdd>b__3_0(String name, ValueTuple`2 arg)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
   at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

I want to confirm that as of the latest release (1.25.9), this issue has indeed been fixed and working in a production scenario.

I was having the same issue on my API for a Blazor WASM project after updating to .net 7. Oddly enough I had updated 2 projects and only 1 was having the issue. The project that was not having the issue makes a single API call on startup before any other calls can be made to gather user permissions. For me and as @magols stated it seems the issue occurs when multiple API calls are made on startup. I updated the 2nd project to make the single service call before any others could be made and it fixed the issue.

@jennyf19 I’ve created a simple project where you can reproduce the problem. There is a single unit test that should pass. https://github.com/f1nzer/ms-identity-web-1957-bug-repro

Using the https://github.com/AzureAD/microsoft-identity-web/tree/jennyf/fix1957 branch I still sometimes get the error from line 214.

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
         at System.Collections.Generic.HashSet`1.Enumerator.MoveNext()
         at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
         at Microsoft.Identity.Web.MergedOptions.UpdateMergedOptionsFromMicrosoftIdentityOptions(MicrosoftIdentityOptions microsoftIdentityOptions, MergedOptions mergedOptions) in C:\Source\microsoft-identity-web\src\Microsoft.Identity.Web.TokenAcquisition\MergedOptions.cs:line 214
         at Microsoft.Identity.Web.MicrosoftIdentityOptionsMerger.PostConfigure(String name, MicrosoftIdentityOptions options) in C:\Source\microsoft-identity-web\src\Microsoft.Identity.Web.TokenAcquisition\OptionsMergers\MicrosoftIdentityOptionsMerger.cs:line 19
         at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
         at Microsoft.Extensions.Options.OptionsCache`1.<>c__3`1.<GetOrAdd>b__3_0(String name, ValueTuple`2 arg)
         at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)
         at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
         at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
         at Microsoft.Identity.Web.MicrosoftIdentityWebAppAuthenticationBuilderExtensions.<>c__DisplayClass5_0.<AddMicrosoftIdentityWebAppInternal>b__3(OpenIdConnectOptions options, IServiceProvider serviceProvider, IOptionsMonitor`1 mergedOptionsMonitor, IOptionsMonitor`1 msIdOptionsMonitor, IOptions`1 msIdOptions) in C:\Source\microsoft-identity-web\src\Microsoft.Identity.Web\WebAppExtensions\MicrosoftIdentityWebAppAuthenticationBuilderExtensions.cs:line 287
         at Microsoft.Extensions.Options.ConfigureNamedOptions`5.Configure(String name, TOptions options)
         at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
         at Microsoft.Extensions.Options.OptionsCache`1.<>c__3`1.<GetOrAdd>b__3_0(String name, ValueTuple`2 arg)
         at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)
         at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
         at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
         at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

Currently running into the same issue, tried the 2.0.5-preview package but the same exception still occurs. Issue only start occurring after upgrading to .NET 7.

@jmprieur We’re still seeing this in multiple APIs, though using “Microsoft.Identity.Web” V2.13.4 Also, we are using .NET 6, only.

You wrote, that’s a regression in .NET 7, but you did not specifically anwser the question of @rondelward-pf, if this fixes it for .NET 6, too .

What confuses me too, is that the Fix was made in “Microsoft.Identity.Web”, so why did you point out the next .NET 7 runtime patch? Shouldn’t the commit fix it by itself?

So I believe the next runtime patch of .NET 7 should fix it.

This is one of these log entries, using V2.13.4

2023-09-21T13:07:16.4158570+02:00 [Thread:] [ERR] (Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer) Connection ID ""5188146776636391632"", Request ID ""800000d1-0001-4800-b63f-84710c7967bb"": An unhandled exception was thrown by the application.
System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value, Int32& location)
   at System.Collections.Generic.HashSet`1.System.Collections.Generic.ICollection<T>.Add(T item)
   at Microsoft.Identity.Web.MergedOptions.UpdateMergedOptionsFromMicrosoftIdentityOptions(MicrosoftIdentityOptions microsoftIdentityOptions, MergedOptions mergedOptions)
   at Microsoft.Identity.Web.MicrosoftIdentityOptionsMerger.PostConfigure(String name, MicrosoftIdentityOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsMonitor`1.<>c.<Get>b__10_0(String name, IOptionsFactory`1 factory)
   at Microsoft.Extensions.Options.OptionsCache`1.<>c__3`1.<GetOrAdd>b__3_0(String name, ValueTuple`2 arg)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
   at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
   at Microsoft.Identity.Web.MicrosoftIdentityWebApiAuthenticationBuilderExtensions.<>c__DisplayClass3_0.<AddMicrosoftIdentityWebApiImplementation>b__2(JwtBearerOptions options, IServiceProvider serviceProvider, IMergedOptionsStore mergedOptionsMonitor, IOptionsMonitor`1 msIdOptionsMonitor)
   at Microsoft.Extensions.Options.ConfigureNamedOptions`4.Configure(String name, TOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at Microsoft.Extensions.Options.OptionsMonitor`1.<>c.<Get>b__10_0(String name, IOptionsFactory`1 factory)
   at Microsoft.Extensions.Options.OptionsCache`1.<>c__3`1.<GetOrAdd>b__3_0(String name, ValueTuple`2 arg)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)
   at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
   at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatcher|8_0(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task`1 matcherTask)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

We believe we have a fix, which we’ll release tomorrow PST. Thanks f1nzer for your repro, which was instrumental.

Unfortunately, I’m still getting the same error using version 1.25.6 but only if there are multiple requests going to the API. If I use Swagger to do a single request the exception does not occur, if I let the frontend do the first requests, the exception occurs twice.

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.Collections.Generic.HashSet`1.Enumerator.MoveNext()                                                                                                                                                                                                                                                                
System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
at Microsoft.Identity.Web.MergedOptions.UpdateMergedOptionsFromMicrosoftIdentityOptions(MicrosoftIdentityOptions microsoftIdentityOptions, MergedOptions mergedOptions)
at Microsoft.Identity.Web.MicrosoftIdentityWebApiAuthenticationBuilderExtensions.<>c__DisplayClass3_0.<AddMicrosoftIdentityWebApiImplementation>b__0(JwtBearerOptions options, IServiceProvider serviceProvider, IOptionsMonitor`1 mergedOptionsMonitor, IOptionsMonitor`1 msIdOptionsMonitor, IOptions`1 msIdOptions)
at Microsoft.Extensions.Options.ConfigureNamedOptions`5.Configure(String name, TOptions options)
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
at Microsoft.Extensions.Options.OptionsCache`1.<>c__3`1.<GetOrAdd>b__3_0(String name, ValueTuple`2 arg)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd[TArg](TKey key, Func`3 valueFactory, TArg factoryArgument)                                                                                                                                                                                                 
at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)   
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context) 

We have the same issue

The linked merged request #1979 is still open and not merged to master yet? And the source code download for the 1.25.9 release doesn’t show it either.

Has this actually not been resolved in the latest nuget release or have I misunderstood something?

1.25.7 is released. We will release the fix in v2-preview as well.

@billyjacobs2014 what do you mean not on a branch? We worked together on it yesterday night with Jenny. We have been prioritizing fixing it, but had not been able to repro it until @magols’s provided clear repro steps

The fix is in jennyf/fix1957. Does the issue still repro when you use packages that you build from that branch? We’d like to have a confirmation that the issue is fixed. We aim to do a new release (1.25.6) today.

Thanks for all the suggestions and repro steps. Does anyone want to see if this branch resolves the issue?

@jennyf19 I have notable problems with this in development of a Serverside Blazor and and mostly when app is restarted during hot reload.

Using the branch https://github.com/AzureAD/microsoft-identity-web/tree/jennyf/fix1957 I still have the issue unfortunately.

I am not well versed in this type of code but I noted that the (!mergedOptions.ClaimActions.Contains(claimAction)) does not behave as seemingly expected as it will always evaluate to false. There is always a call to Add() duplicates into mergedOptions.ClaimActions.

As illustrated by this screenshot with duplicate DeleteClaimActions with the same ClaimType and ValueType.