runtime: Exception in test infrastructure - System.InvalidOperationException: Collection was modified after the enumerator was instantiated
For the second time I have encountered the following error during CI OSX test run.
https://ci.dot.net/job/dotnet_coreclr/job/master/job/x64_checked_osx10.12_innerloop_tst_prtest/5550/
02:51:26 Unhandled Exception: System.InvalidOperationException: Collection was modified after the enumerator was instantiated.
02:51:26 at System.Collections.Generic.Stack`1.Enumerator.MoveNext()
02:51:26 at Xunit.Sdk.DisposalTracker.Dispose() in C:\projects\xunit\src\xunit.execution\Sdk\DisposalTracker.cs:line 26
02:51:26 at Xunit.Sdk.ExtensibilityPointFactory.Dispose() in C:\projects\xunit\src\xunit.execution\Sdk\ExtensibilityPointFactory.cs:line 54
02:51:26 at Xunit.Sdk.TestFramework.Dispose() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\TestFramework.cs:line 49
02:51:26 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
02:51:26 --- End of stack trace from previous location where exception was thrown ---
02:51:26 at System.Threading.ThreadPoolWorkQueue.Dispatch()
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 85 (75 by maintainers)
According to runfo this isn’t happening anymore in main or release/6.0: https://runfo.azurewebsites.net/tracking/issue/111/
As the fix was merged into release/6.0 but not release/6.0-rc2 there are still a few hits for the RC2 branch but those will just naturally go away when RC2 doesn’t produce any builds anymore.
Closing.
Woohoo
The issue (in v2) is this:
We support running multiple assemblies in parallel, but the only runners that officially support this are the .NET Framework runners (our console and MSBuild runners). There is app domain isolation available there that helps prevent multiple test assemblies from stepping on each other, though there are edge cases inside .NET Framework itself that can still confound (like the fact that “current directory” is a process-wide setting, so calling
Directory.SetCurrentDirectoryaffects all app domains in the process; by and large, app domain isolation does all the isolation heavy lifting).Most teams–I want to say “everybody except the .NET team” but I don’t have the data to back that up–who run .NET Core unit tests use the third party
dotnet testfunctionality (or something that consumes that same plugin, like Visual Studio and VS Code). Whether those things support running multiple assemblies in parallel is up to them, and how they deal with isolation is up to them (the only reasonable option they’d have is process isolation, I assume). We don’t provide that functionality because it’s outside the scope of the VSTest plugin API.What the .NET team appears to be doing is using a forked version of a deprecated runner that was a .NET Core version of our console runner (originally designed for the now removed
dotnet xunitfunctionality), which is attempting to run multiple assemblies in the same address space. .NET Core does not have app domain isolation (or any other isolation, for that matter) that can help prevent the kinds of collisions that you’ve hit. This is why the bug your team is hitting is only being hit by you: because you’re the only people trying to write your own runner in this way (or if anybody else is doing so, they’ve fixed or ignored the problem in their own way without raising it with xUnit.net).As for v3, the entire design of test projects has changed from “compiles to library” to “compiles to application”. The issues with isolation and dependency resolution in .NET Core were a big part of the forcing function. In v3, our supported console runner will allow you to run .NET Core tests in addition to .NET Framework tests (and even parallelize .NET Framework and .NET Core tests together, as well as having significantly better support for varying versions of .NET Framework or .NET Core, allow running 32- and 64-bit tests in parallel, etc.).
Even though the failure occurs in runtime tests most of the time (which use the xunitwrapper infra) there are also hits in libraries tests.
@maryamariyan is actually looking at fixing that bug, and as part of that fix we’d like to describe why it’s happening. Xunit has a static API that will operate on this shared stack, so something is calling that static API from multiple threads. We were going to trace a test exectuion to see why that happens from multiple threads. To find a good test to trace we started looking at the existing repros and that’s when we noticed this pattern. It seemed too curious not to ignore.
We’d also like to make sure that a fix for this
InvalidOperationExceptiondoesn’t just result in exposing a new exception because we disposed some shared state that the other thread needs. If we can identify root cause it would be good.