stryker-net: Negative value specified for TestSessionTimeout - Failed to create a TestRunner, Stryker cannot recover from this

Describe the bug

I got some warnings like this: Runner 1: Some mutations were executed outside any test It then looks like it got a bit of the way through (2.47%) then blew up with:

*** Failed to create a TestRunner, Stryker cannot recover from this!***


   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList`1 list, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body)
   at Stryker.Core.MutationTest.MutationTestProcess.TestMutants(IEnumerable`1 mutantsToTest) in /_/src/Stryker.Core/Stryker.Core/MutationTest/MutationTestProcess.cs:line 147
   at Stryker.Core.MutationTest.MutationTestProcess.Test(IEnumerable`1 mutantsToTest) in /_/src/Stryker.Core/Stryker.Core/MutationTest/MutationTestProcess.cs:line 131
   at Stryker.Core.StrykerRunner.RunMutationTest(IStrykerInputs inputs, ILoggerFactory loggerFactory, IProjectOrchestrator projectOrchestrator) in /_/src/Stryker.Core/Stryker.Core/StrykerRunner.cs:line 107
   at Stryker.CLI.StrykerCli.RunStryker(IStrykerInputs inputs) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 93
   at Stryker.CLI.StrykerCli.<>c__DisplayClass10_0.<Run>b__0() in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 68
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass143_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Stryker.CLI.StrykerCli.Run(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 74
   at Stryker.CLI.Program.Main(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/Program.cs:line 14

Logs A huge amount of stuff is in those logs. I’d have to look through more carefully before just uploading them. The phrase “a TestRunner” doesn’t appear in any of them, so I’m not sure how much they’ll help.

Expected behavior Keeps running to completion. Exits with output based on tests successfully completed.

Desktop (please complete the following information):

OS: Windows 11
Type of project: Net 6
Stryker Version: 1.5.3
Runtime/sdk version: 6.0.300-preview.22154.4

Additional context

Thanks for all the work on this project, I’ve been trying it occasionally over the years and this is by far the easiest to setup and closest I’ve got it to working yet. Sorry if there’s a doc link for this sort of error, Google didn’t find it

Probably a separate issue

For my use case, even if there is some issue, ideally it’d keep going. Luckily I ran it restricted to changes since a recent commit, and checked first, but I was going to leave it running overnight since it predicted 5 hours for the full run, and it’d be a bit sad if it failed after 4 hours with absolutely no output. Is there some way to do incremental output, or automatically/manually “run again, but skip the thing that blew up last time”?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (8 by maintainers)

Most upvoted comments

I am not observing any failures or errors in the testrunners when running against your reproduction and I am getting timeouts on some static mutants (probably due to changed timeout average with these new randomly generated tests) so I am gonna go ahead and mark this fixed for now 😃

rouke-broersma on Aug 5, 2022

#2039 was intended to fix it. There’s a repro above if someone wants to retest on a particular version https://github.com/stryker-mutator/stryker-net/issues/2069#issuecomment-1147337889

GrahamTheCoder on Aug 5, 2022

While redesigning the code, I encountered several issues, which were hidden being the duration re computation logic. Having added dedicated unit tests, I am confident we can reach a situation where we trustfully process the time reported by the runners.

dupdob on Jun 13, 2022

Yeah, I understand that knowing about individual different issues is important and things that don’t cause breakage tend to go unreported. I thought of the command line option initially, but it seemed like quite a bit of work to still get what’s still not very good solution: e.g. User experiences potentially transient failure -> read the docs -> tendency to use the option everywhere from now on, still masking the issues.

That may be a risk, but I fail to see how hiding it preemptively is better. And there are strategies to prevent this kind of overgeneralization. But I suggest it would better to discuss in a separate topic and stay on track here.

Any chance I can access this project somehow ? At this stage this is either a new issue in timeout calculation in Stryker (as when we discovered the hard way how xUnit was lying about it), or that NUnit duration reports are invalid. But is seems hard to diagnose without a reproduction scenario

dupdob on May 31, 2022

hi, thanks for digging into this

I wondered if it’s something like a shared setup method being counted in each of many testcases (since I have a lot of sub testcases), so I changed this to the xUnit impl (not sure if that covers what I’m talking about), but still no luck:

What did you change or how did you change it ? I think I see the current code version; I would love to see how you tried to fix this.

My tests are NUnit. I noticed the code only disabled paralleization for xUnit. I did try adding that line for NUnit too but it didn’t seem to change the runtime or fix the issue:

Actually, it does disable parallelization via the CollectDataForEachTestSeparately setting (see NUnit source code here).

In terms of unblocking the use of stryker, would you accept a PR that logs a warning if it’s negative and sets the value calculated here to some default value (e.g. 0): Could you open an issue for this ? I had a similar idea, but more along the lines of adding a CLI options that would disregard errors and attempt to go through the whole process anyway. This could unblock users while still providing valuable feedback to the project. Timing computations is important as it impacts the total runtime, so we need to fix each bug you find.

I wondered if it’s something like a shared setup method being counted in each of many testcases (since I have a lot of sub testcases).

Could be… NUnit aggregates all results for a given testcase in a single VsTest result (while xUnit reports each one separately), so the xUnit logic does not make any sense for NUnit.

Are you using the lates NUnit version ? so we cab check if this is a NUnit issue.

dupdob on May 31, 2022