nunit: Parametrized testcases not running in parallel

Issue

I am trying to run several cases of a parametrized test in parallel. Here is the code I’m using:

class Dummy
{
    static TestCaseData Case(int i)
        => new TestCaseData(TimeSpan.FromSeconds(2)).SetName($"Case {i}");

    public static IEnumerable<TestCaseData> Cases()
        => Enumerable.Range(1, 5).Select(Case);

    [TestCaseSource(nameof(Cases)), Parallelizable]
    public void ItShouldSleep(TimeSpan t)
        => Thread.Sleep(t);
}

When I run this using the Visual Studio test runner, I can see the individual cases being executed one at a time, and the overall test run takes ~10 seconds. I would expect all the cases to be executed simultaneously on separate threads, and for the overall run to take ~2 seconds.

Setup

I am using the following .NET Standard based csproj file alongside the code above:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Library</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <ApplicationIcon />
    <StartupObject />
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
    <PackageReference Include="NUnit" Version="3.8.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.8.0" />
  </ItemGroup>

</Project>

NuGet packages:

Microsoft.NET.Test.Sdk: v15.3.0
Microsoft.NETCore.App: v2.0.0
NUnit: v3.8.1
NUnit3TestAdapter: v3.8.0

Visual Studio:

NUnit 3 Test Adapter: 3.8.0.0
Visual Studio: 15.3.5

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 29 (17 by maintainers)

Commits related to this issue

Fix for https://github.com/nunit/nunit/issues/2471 — committed to rprouse/NUnit.Parallelism.Integration.Tests by rprouse 7 years ago

Most upvoted comments

@DrunkRussianGun

FYI, I no longer work on the NUnit framework. But I’ll try to wrap up my comments on this since you mentioned me…

This is a very old (2017) issue on an old version of the framework. My comments, made in 2018, are about that version only.
The user closed the issue because he discovered an error in his own code. No changes were made to NUnit.
It’s not likely that you will get a lot of help by piling on to an old, closed issue, caused by a different problem.
The example you give is very clear and would be good for creating a new issue, which the current team could work.

Just saying. 😄

WRT your example, I suggest simplification where possible. Don’t use any attributes if you want default (non-parallel) behavior. Try to avoid setting up parallelism at a higher level and then removing it at a lower level. IOW, rather than setting ParallelScope.Children or All on the fixture and removing it on some child tests, Pick out the tests you want to run in parallel and use the Parallelizable attribute without any argument. Doing it that way is (1) easier for the reader to follow and (2) exercises the best tested path within the framework. Also, you may find that doing it one way works while a different way breaks things.

You don’t say how you are running tests. For the purpose of showing a bug, i suggest use of NUnitLite. That’s because it leaves out any complications caused by the engine, console runner or VS adapter and gives the team a very simple way to replicate it.

Good luck!

CharliePoole on Oct 6, 2020

Not I can answer on @rprouse behalf, but I think that work on a .NET Standard 2.0 version will begin shortly (after the smaller tasks - due to the major cleanup in #2380 - has been done). Please track this issue #2542.

mikkelbu on Nov 28, 2017

I’ve determined that [Parallelizable(ParallelScope.Children)] works correctly on parameterized methods in the latest build of #2476, so I’m counting this as being fixed when that PR is merged.

We have a general set of problems around attributes that appear on a .test method but which are expected to somehow affect all the test cases for that method.

There are three potential situations

Attributes that specifically apply to the cases
Attributes that do not apply to the cases
Attributes that do not apply to the cases but still impact them in some way (e.g. IgnoreAttribute causes the parameterized test method suite to be ignored, resulting in the cases being ignored as well even though they don’t have that setting.)

We may want to address this problem more generally at some point. For this issue, we are only looking at the ParallelizableAttribute and NonParallelizable attribute.

Starting with [NonParallelizable]… this one is unambiguous. It sets the method to be non-parallelizable as well as the context in which the cases are run. Nothing is parallelized once this attribute is used.

[Parallelizable] used with a scope is also pretty clear.

If the scope is Self, then the method may run in parallel with other methods of the same class. The individual cases under the method are run sequentially.
If the scope is Children then the method may not run in parallel with other methods but all it’s cases may run in parallel with one another.
If the scope is All then the cases may run in parallel with one another as well as with other parallel methods.

If one remembers that [Parallelizable] is equivalent to [Parallelizable(ParallelScope.Self)], then this case is also clear. However, it seems to be very common for users to mistake the meaning in this case. Our choices are:

Make the bare attribute mean something else when it’s applied to a parameterized method. This seems like a potential source of even greater confusion to me.
Make using [Parallelizable] on a parameterized method an error.
Give a warning message of some kind when [Parallelizable] is used on a parameterized method, in order to encourage the user to provide a ParallelScope argument.
Just improve the documentation.

I prefer approach number 4. What do others think?

CharliePoole on Oct 29, 2017

I reopened this to look at both attribute validation and the handling of parameterized methods.At very least, I’ll add it to the set of parallel tests I’m creating.

CharliePoole on Oct 6, 2017

Issue here was that I wasn’t applying the attribute correctly. You need to have [Parallelizable(ParallelScope.Children)] on the containing class.

masaeedu on Oct 6, 2017

@nunit/framework-team I updated the usage note and the descriptions of all three parallelism-related attributes in the docs.

@masaeedu Check to see which build of the framework is referenced in your test. If it’s .NET Standard or (for older NUnit) Portable, then you get no parallelism. This is not exactly an issue for us to track. We can’t support parallelism in either the .NET standard 1.3 or 1.6 build because they don’t provide the support we would need for it. We have an issue for creation of a .NET standard 2.0 build, which would support it.

CharliePoole on Oct 4, 2017