nunit: NUnit.Framework 3.13.2 introduced a breaking change that conceals problems with tests

Before NUnit.Framework 3.13.2 running an assembly compiled from the following code

[TestFixture]
public class BrokenTestClass
{
  [Test, Invalid] public void Test() { }
}

public class InvalidAttribute : Attribute
{
  public InvalidAttribute() => throw new InvalidProgramException();
}

would issue the An exception was thrown while loading the test twice:

• once inside the xml returned from ITestEventListener.Explore,
• and the second time inside a <test-suite type="TestFixture" ... /> event for NUnit.Engine.ITestEventListener.

Since 3.13.2 this error is only issued during discovery, and is not issued during execution.

The new behavior broke R#/Rider test runner in that a broken test class no longer gets a clear indication of a problem, marking such test as inconclusive instead.

More importantly, the new behavior broke NUnit’s own runner in that it no longer reports such errors, making problems of this nature very difficult to notice. For this reason over a 1000 tests weren’t running in our CI over the past week without us knowing.

Below is the output of nunit3-console running an assembly with broken tests:

Copyright (c) 2022 Charlie Poole, Rob Prouse
Wednesday, April 13, 2022 3:49:53 PM

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    c:\Users\jetbrains\source\repos\Tests.Sandbox\Sandbox.NUnit3\bin\Debug\net35\Sandbox.NUnit3.dll


Run Settings
    DisposeRunners: True
    WorkDirectory: C:\Users\jetbrains\Documents\59\bin\net35
    ImageRuntimeVersion: 2.0.50727
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    TargetRuntimeFramework: net-2.0
    NumberOfTestWorkers: 16

Test Run Summary
  Overall result: Passed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2022-04-13 13:49:53Z
    End time: 2022-04-13 13:49:53Z
    Duration: 0.510 seconds

Results (nunit3) saved as TestResult.xml