stryker-net: Stryker.NET doesn't handle xUnit v3 properly
Describe the bug
When using xUnit v3, Stryker.NET produces warnings and can’t perform proper mutations.
Reproduction steps
Checkout the xunit-v3 branch of the Serilog.Formatting.Log4Net project and run dotnet tool restore && dotnet stryker at the root of the repository.
Expected behavior
Stryker.NET performs mutations and gives the same score (100%) as when using xUnit v2.
Dashboard with xUnit v2 (on the main branch): 100%
Actual behaviour
Stryker.NET produces many warnings and eventually fails to perform the mutations.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.IndentationSettingsTest.IndentationSettingsToString), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.ThreadIdProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.MessageCDataMode), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.IndentationSettings), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.DomainAndUserNameProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.LogEventLevel), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.MachineNameProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.XmlElementsLineEnding), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.Log4JCompatibility), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.PublicApi.ApprovePublicApi), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
Dashboard with xUnit v3: 3.09%
Desktop
- OS: both macOS and Linux (and probably Windows too)
- Type of project: .NET 6 + 8 + .NET Standard 2.0
- Framework Version .NET 8
- Stryker Version: 4.4.1
Additional context
I have not enabled Microsoft Testing Platform support in xUnit.net v3 so I don’t think this is related to #3094 although I’m not totally sure.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 16 (13 by maintainers)
Commits related to this issue
- revert: xunit v3 migration (#18) Revert back the xunit v3 migration from #14, as [Stryker.NET doesn't handle xUnit v3 properly](https://github.com/stryker-mutator/stryker-net/issues/3117). — committed to aweXpect/aweXpect.Chronology by vbreuss 2 years ago
- Remove .NET Stryker mutation badge, because [Stryker.NET doesn't handle xUnit v3 properly](https://github.com/stryker-mutator/stryker-net/issues/3117) — committed to Testably/Testably.Abstractions by vbreuss 2 years ago
- refactor: disable explicit .NET7 and upgrade to xunit v3 (#728) Upgrade testing projects to [xunit v3](https://xunit.net/docs/getting-started/v3/migration) and remove the .NET Stryker mutation badge,... — committed to Testably/Testably.Abstractions by vbreuss 2 years ago
- revert: xunit v3 migration Revert back the xunit v3 migration, as [Stryker.NET doesn't handle xUnit v3 properly](https://github.com/stryker-mutator/stryker-net/issues/3117). — committed to aweXpect/aweXpect.Web by vbreuss 2 years ago
- revert: xunit v3 migration Revert back the xunit v3 migration, as [Stryker.NET doesn't handle xUnit v3 properly](https://github.com/stryker-mutator/stryker-net/issues/3117). — committed to aweXpect/aweXpect.Json by vbreuss 2 years ago
- revert: xunit v3 migration (#19) Revert back the xunit v3 migration, as [Stryker.NET doesn't handle xUnit v3 properly](https://github.com/stryker-mutator/stryker-net/issues/3117). — committed to aweXpect/aweXpect.Web by vbreuss 2 years ago
- revert: xunit v3 migration (#26) Revert back the xunit v3 migration, as [Stryker.NET doesn't handle xUnit v3 properly](https://github.com/stryker-mutator/stryker-net/issues/3117). — committed to aweXpect/aweXpect.Json by vbreuss 2 years ago
- Use xUnit v3 through Microsoft Testing Platform Remaining issues * GitHub action: how to replace GitHubActionsTestLogger? See https://github.com/microsoft/testfx/issues/4365 * HTML logger: could not... — committed to serilog-contrib/serilog-formatting-log4net by 0xced 2 years ago
- Use xUnit v3 through Microsoft Testing Platform Remaining issues * GitHub action: how to replace GitHubActionsTestLogger? See https://github.com/microsoft/testfx/issues/4365 * HTML logger: could not... — committed to serilog-contrib/serilog-formatting-log4net by 0xced 2 years ago
- Use xUnit v3 through Microsoft Testing Platform Remaining issues * GitHub action: how to replace GitHubActionsTestLogger? See https://github.com/microsoft/testfx/issues/4365 * HTML logger: could not... — committed to serilog-contrib/serilog-formatting-log4net by 0xced 2 years ago
Let’s ping @bradwilson on this issue. He’s been very helpful when I reported issues on the early xUnit v3 prereleases. Maybe he can give some guidance.
PR #3122 restore basic support for xUnit V3. Basic as no coverage based optimization possible.
Consequences
I do not think trying to fit support for xUnitV3 via VsTest is worth the effort, I suppose the move to mstest will provide a smoother solution. But that remains to be confirmed.
The alternative is to implement a dedicated runner for xUnit V3, which would provide other interesting benefits but looks like a lot of work.
I predict a big one: it looks like a lot of VsTest design rules are violated here.