vstest: dotnet test is significantly slower compared to what was available with project.json.

Remove extra appdomain creation for getting source info using DiaSession

This can’t be done without breaking VS <= 2015.

Batching to reduce cross-appdomain communication

Can you explain this?

Cost due to conversion of XUnitTestCase to VSTestCase (MarshalByRef object)

The Visual Studio runner (on which dotnet test is based on now in MSBuild-land, but wasn’t previously in project.json-land) has always been agonizingly slow compared to every other environment. The multi-process design, with the need to serialize test cases, is just brutal for performance. We have actively steered people away from using it pretty much since we shipped 2.0. The fact that it’s at the core of dotnet test is unfortunate in this respect, and it seems unlikely that you’re going to be able to fix what is fundamentally a design flaw that dates back to 2012.

Our solution for this is: We intend to write a new dotnet command (dotnet xunit), and once that’s available, tell people to stop using dotnet test. This will solve not only the poor performance issues, but also the fact that dotnet test is nowhere near as usable or feature rich as our other custom runners (as well as stops forcing people to opt into the Test Explorer integration even if all they cared about is command line test execution).

We will not be making another RTM release before March 7th.

Here are the numbers after applying the serialization optimizations and designmode support in test platform (https://github.com/xunit/visualstudio.xunit/pull/103 and https://github.com/Microsoft/vstest/pull/520):

Notes:

• All numbers are with dotnet test --no-build
• For no appdomain, use dotnet test -- RunConfiguration.DisableAppDomain=true with csproj projects. This mode was default in dotnet-test-xunit (version used in repro steps).

I took a look with PerfView and noticed a number of problems.

1. I couldn’t actually get the netcoreapp1.1 tests to run. These kept failing for me with “Error: Failed to initialize client proxy: could not connect to test process.” I assume there’s some bits cobbled on my machine causing this failure, so I didn’t spend anytime investigating. Instead, I just passed -f net451 to dotnet test.
2. As noted by others, build takes a big chunk of the time. Passing --no-build addresses this.
3. The tests do run slower, and it appears that a lot of this is in the xUnit test adapter itself.

dotnet test --no-build -f net451 = 7.95 seconds

From PerfView, it looked like another chunk of time was being spent in cross-AppDomain communication. Knowing that xUnit will use AppDomains if they’re availabe, I added an XML config file next to the test dll named ‘Microsoft.AspNetCore.Authentication.Test.dll.config’ with the following content.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <appSettings>
    <add key="xunit.appDomain" value="denied"/>
  </appSettings>
</configuration>

That shaved off a ~1/3 of the time. The difference in the speed of the console output was visually noticable.

dotnet test --no-build -f net451 = 5.25 seconds

That’s at least getting closer.

Have you tried using the /Parallel flag to see if it affects performance? I’ve tried but it doesn’t appear to change perf. e.g. dotnet test -- /Parallel.

Side note: I’m not sure this is the right syntax. It shows up in dotnet-vstest help output.

We’re seeing about a 10x increase in test execution time in Kestrel.