nunit: Dangerous behaviour of TimeoutAttribute on NET Core and later

Due to the removal of Thread.Abort in later versions of .NET, the TimeoutAttribute no longer stops the long running test.

This means the test keeps running in the background, despite that the nunit runner moves to the next test. As an example, see the following testfixture:

internal class TimeOut
{
    private string _logFilePath;

    [OneTimeSetUp]
    public void OneTimeSetup()
    {
        _logFilePath = Path.Combine(TestContext.CurrentContext.TestDirectory, "Timeout.txt");
        File.Delete(_logFilePath);
    }

    private void WriteLine(string text)
    {
        File.AppendAllText(_logFilePath, text + Environment.NewLine);
    }

    [TearDown]
    public void Teardown()
    {
        WriteLine("Teardown, result: " + TestContext.CurrentContext.Result.Outcome);
    }

    [Test]
    [Order(1)]
    [Timeout(110)]
    public void WithTimeout()
    {
        LongTest("WithTimeout");
    }

    [Test]
    [Order(2)]
    public void NoTimeout()
    {
        LongTest("NoTimeout");
    }

    private void LongTest(string testName)
    {
        try
        {
            for (int i = 0; i < 5; i++)
            {
                WriteLine($"{testName}: Iteration {i}");
                Thread.Sleep(100);
            }
            Assert.Pass("Success");
        }
        catch (Exception ex)
        {
            WriteLine(ex.Message);
        }
        finally
        {
            WriteLine("Task finished");
        }
    }
}

When running this on NET48, the `net48/Timeout.txt’ file contains:

WithTimeout: Iteration 0
WithTimeout: Iteration 1
Thread was being aborted.
Task finished
Teardown, result: Failed:Cancelled
NoTimeout: Iteration 0
NoTimeout: Iteration 1
NoTimeout: Iteration 2
NoTimeout: Iteration 3
NoTimeout: Iteration 4
Task finished
Teardown, result: Passed

This shows the 1st test being aborted, and its Teardown method running, before the 2nd test starts.

When running on .NET5, the net5.0/Timeout.txt contains:

WithTimeout: Iteration 0
WithTimeout: Iteration 1
NoTimeout: Iteration 0
WithTimeout: Iteration 2
NoTimeout: Iteration 1
WithTimeout: Iteration 3
NoTimeout: Iteration 2
WithTimeout: Iteration 4
NoTimeout: Iteration 3
Task finished
NoTimeout: Iteration 4
Task finished
Teardown, result: Passed

This shows that the WithTimeout test runs to the end but with no Teardown running. At the same time, the 2nd test is running in parallel, starting after the timeout of the 1st test.

There are multiple issues:

1. TearDown is not run when the test ‘times out’
2. The test keeps running, which on tests with shared instance fields could be problematic, possibly interfering with other tests.

If I add a _counter field, set to 0 in SetUp, increment it in the for-loop and Assert.That(_counter, Is.EqualTo(5));, in the actual test, the net5.0 run fails with:

 Failed NoTimeout [507 ms]
  Error Message:
     Expected: 5
  But was:  8

  Stack Trace:
     at Tests.TimeOut.NoTimeout() in C:\Users\m.brands\source\Tests\TimeOut.cs:line 53

The question is what can be done about it.

1. Mark the TimeoutAttribute as obsolete except for the build targeting net framework?
2. Add an analyzer rule in nunit.analyzers warning about the drawbacks on non-framework builds.
3. Can we fix the behaviour to what it was before. Not likely.
4. Add a CancellationToken to TestContext, set to either CancellationToken.None if no timeout set, or a token that will be cancelled after the time out, see below. This approach will require updating end-users tests to ‘the new way’.

using CancellationTokenSource cancellationTokenSource = new CancellationTokenSource(110);
await LongTest("WithTimeout", cancellationTokenSource.Token);
Assert.That(_counter, Is.EqualTo(5));