stryker-net: Stryker.NET doesn't handle xUnit v3 properly

Describe the bug

When using xUnit v3, Stryker.NET produces warnings and can’t perform proper mutations.

Reproduction steps

Checkout the xunit-v3 branch of the Serilog.Formatting.Log4Net project and run dotnet tool restore && dotnet stryker at the root of the repository.

Expected behavior

Stryker.NET performs mutations and gives the same score (100%) as when using xUnit v2.

Dashboard with xUnit v2 (on the main branch): 100%

Actual behaviour

Stryker.NET produces many warnings and eventually fails to perform the mutations.

[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.IndentationSettingsTest.IndentationSettingsToString), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.ThreadIdProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.MessageCDataMode), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.IndentationSettings), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.DomainAndUserNameProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.LogEventLevel), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.MachineNameProperty), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.XmlElementsLineEnding), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.Log4NetTextFormatterTest.Log4JCompatibility), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.
[22:03:18 WRN] Runner 2: Initial test run encounter an unexpected test case (Serilog.Formatting.Log4Net.Tests.PublicApi.ApprovePublicApi), mutation tests may be inaccurate. Disable coverage analysis if you have doubts.

Dashboard with xUnit v3: 3.09%

Desktop

  • OS: both macOS and Linux (and probably Windows too)
  • Type of project: .NET 6 + 8 + .NET Standard 2.0
  • Framework Version .NET 8
  • Stryker Version: 4.4.1

Additional context

I have not enabled Microsoft Testing Platform support in xUnit.net v3 so I don’t think this is related to #3094 although I’m not totally sure.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 16 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Let’s ping @bradwilson on this issue. He’s been very helpful when I reported issues on the early xUnit v3 prereleases. Maybe he can give some guidance.

PR #3122 restore basic support for xUnit V3. Basic as no coverage based optimization possible.

Consequences

  1. Stryker does not currently support xUnitV3 test adapter as it deviates too much from the standard behavior
  2. The main consequence is that Stryker cannot work AT ALL with xUnitV3 test base. Any attempt will result in most mutants being flag as not covered and the remaining flagged as survived.
  3. There is no clear path forward to fix this
  4. In the short term, Stryker can be modified to support a simpler approach to tests. That is one without coverage analysis and using environment variables for control. Note that it will result in very poor performances.
  5. For medium term, some basic form of coverage could be restored (is a mutation covered yes or no), that would only slightly improve performances
  6. Alternatively, we could revert to using IPC mechanisms to exchange data between coverage collector and the testing logic. But bear in mind this logic was dropped in the pas as it was complex and a recurrent source of issues.

I do not think trying to fit support for xUnitV3 via VsTest is worth the effort, I suppose the move to mstest will provide a smoother solution. But that remains to be confirmed.

The alternative is to implement a dedicated runner for xUnit V3, which would provide other interesting benefits but looks like a lot of work.

I predict a big one: it looks like a lot of VsTest design rules are violated here.