nunit3-vs-adapter: dotnet test with category filter is slow with a lot of tests

When there are thousands of tests, filtering by category takes a huge amount of time.

I noticed this with Noda Time when trying to migrate from NUnitLite to dotnet test, but I’ve now reproduced separately.

Project file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.6.2" />
    <PackageReference Include="NUnit" Version="3.10.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.10.0" />
  </ItemGroup>
</Project>

I have 500 generated test classes, each of which has 40 tests in, like this:

using NUnit.Framework;
public class GeneratedTest0
{
    [Test] public void Test1() { }
    [Test] public void Test2() { }
    [Test] public void Test3() { }
    [Test] public void Test4() { }
    ...
}

(I’m happy to provide the generator code if that would help.)

When running the tests without a filter, they’re reasonably quick:

dotnet test -c Release
...
Total tests: 20000. Passed: 20000. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 13.9321 Seconds

When a filter is specified, it slows down massively - from 14 seconds to over 110:

dotnet test -c Release --filter=TestCategory!=Slow
...
Total tests: 20000. Passed: 20000. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.9212 Minutes

An equivalent test with xUnit didn’t show the same issue, so I don’t think it’s a problem in dotnet test itself. (I’m working on running this through a profiler to see.)

Version information for dotnet:

dotnet --info
.NET Command Line Tools (2.1.103)

Product Information:
 Version:            2.1.103
 Commit SHA-1 hash:  60218cecb5

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.103\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.6
  Build    : 74b1c703813c8910df5b96f304b0f2b78cdf194d

I have VS15.6.4 installed, if that’s relevant.