nunit: AppVeyor Test Failures under .NET 3.5

These test failures under .NET 3.5 are getting consistent enough that we are getting few passing builds and we had one build time out at 60 minutes in the .NET 3.5 tests.

In PR #2380 I am going to #if net35 and increase the timeout as a temporary workaround, but we should look at this. I am curious why we are having so many .NET 3.5 threading issues, but no other target.

1) Failed : NUnit.Framework.Internal.Execution.EventQueueTests+SetWaitHandle_Enqueue_AsynchronousTest.SetWaitHandle_Enqueue_Asynchronous
Test exceeded Timeout value of 1000ms

1) Failed : NUnit.Framework.Internal.Execution.EventQueueTests+DequeueBlocking_StopTest.DequeueBlocking_Stop
Test exceeded Timeout value of 1000ms

7) Cancelled : NUnit.Framework.Internal.Execution.EventQueueTests+SetWaitHandle_Enqueue_AsynchronousTest.SetWaitHandle_Enqueue_Asynchronous
Test cancelled by user
   at NUnit.Framework.Internal.Reflect.InvokeMethod(MethodInfo method, Object fixture, Object[] args) in C:\projects\nunit\src\NUnitFramework\framework\Internal\Reflect.cs:line 232
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunNonAsyncTestMethod(TestExecutionContext context) in C:\projects\nunit\src\NUnitFramework\framework\Internal\Commands\TestMethodCommand.cs:line 109
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context) in C:\projects\nunit\src\NUnitFramework\framework\Internal\Commands\TestMethodCommand.cs:line 64
   at NUnit.Framework.Internal.Commands.BeforeTestCommand.Execute(TestExecutionContext context) in C:\projects\nunit\src\NUnitFramework\framework\Internal\Commands\BeforeTestCommand.cs:line 50
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.Execute(TestExecutionContext context) in C:\projects\nunit\src\NUnitFramework\framework\Internal\Commands\BeforeAndAfterTestCommand.cs:line 59

1) Failed : NUnit.Framework.Assertions.CollectionAssertTest.PerformanceTests(System.Linq.Enumerable+<RangeIterator>d__b1)
Elapsed time of 137.7538ms exceeds maximum of 100ms

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 29 (29 by maintainers)

Commits related to this issue

Increase timeout in .NET 3.5 queue/dequeue tests. See #2579 — committed to nunit/nunit by rprouse 7 years ago

Most upvoted comments

I’ve started looking at this. First step is to start reproducing locally.

Notes so far

Running .NET 3.5 tests directly on my laptop

I cannot reproduce just running the .NET 3.5 tests on my machine. Best guess is that it is related to number of cores? Pretty sure that the AppVeyor VMs have one core, so we default to two workers to run. I’ve tried locally with one and two workers, but no luck.

Running tests in a .NET 3.5 docker container

Next test, creating a docker image with this Dockerfile,

FROM microsoft/dotnet-framework:3.5
WORKDIR \app
COPY tools/nunit.consolerunner.3.7.0/NUnit.ConsoleRunner/tools/* ./
ENTRYPOINT ["nunit3-console.exe", "net35/nunit.framework.tests.dll"]

Which I build with

docker build -t rprouse/nunit .

First build is slow as it pulls all of the base images needed. Subsequent are much quicker. I can then run the tests with,

docker run -it -v c:\src\nunit\nunit\bin\Release\net35:c:\app\net35 rprouse/nunit

Number of workers is correct, but 10 runs succeeded 😦

Run multiple docker instances simultaneously

Run the .NET 3.5 tests in three docker containers at the same time to put the system under load?

Woot, got it two of the three test runs failed.

1) Failed : NUnit.Framework.Internal.Execution.EventQueueTests+DequeueBlocking_StopTest.DequeueBlocking_Stop
Test exceeded Timeout value of 2000ms

and

1) Failed : NUnit.Framework.Internal.Execution.EventQueueTests+SetWaitHandle_Enqueue_AsynchronousTest.SetWaitHandle_Enqueue_Asynchronous
Test exceeded Timeout value of 2000ms

It isn’t failing every time, but getting closer…

rprouse on Feb 27, 2018

Actually, in https://ci.appveyor.com/project/CharliePoole/nunit/build/3.10.0-ci-04960-pr-2586#L318 3.5 is failing with

1) Failed : NUnit.Framework.Assertions.CollectionAssertTest.PerformanceTests(System.Linq.Enumerable+<RangeIterator>d__b1)
Elapsed time of 137.7538ms exceeds maximum of 100ms

but 4.5 is also failing (in another test)

1) Failed : NUnit.Framework.Assertions.CollectionAssertTest.PerformanceTests(System.Collections.Generic.List`1[System.Double])
Elapsed time of 118.375ms exceeds maximum of 100ms

mikkelbu on Dec 7, 2017

This isn’t fully fixed, we had a test hang in https://ci.appveyor.com/project/CharliePoole/nunit/build/3.10.0-ci-05316-issue-2405. There have been two other failures, but they don’t have the new code because the fork wasn’t updated. It is looking promising for some of the callstacks we’ve seen.

I am leaving open and moving to the 3.11 milestone.

rprouse on Mar 10, 2018

Came here to say the same as @mikkelbu.

jnm2 on Dec 7, 2017

I was thinking of using the 3.5 Docker container microsoft/dotnet-framework:3.5 to run and maybe debug our 3.5 tests repeatedly to try to track it down and see if it happens.

rprouse on Dec 7, 2017

I could be wrong but I haven’t seen any of our timeouts measure slowness of code, only binary correctness of logic.

It would be nice to make it a standard that Timeout is only for binary logic failures (simply as a backstop so that we don’t wait an hour for CI to fail) and MaxTime for performance measurements (because we’ll have actual numbers rather than interrupting it.)

In some cases, you might even use them together but each for a separate purpose.

jnm2 on Dec 2, 2017