vstest: Unable to run/debug XUnit tests targeting Desktop from Visual Studio on latest Windows 10 machine

Description

After the Windows 10 Creators update, I can no longer run or debug xunit tests targeting Desktop .NET Framework in Visual Studio.

Steps to reproduce

1. On a machine with .NET 4.7 and VS 2017 15.2. I think .NET 4.7 comes with Windows Creators update. After I installed .NET 4.7 on another machine that had .NET 4.6, I don’t see System.Runtime, Version=4.1.0.0 in GAC. The version 4.0.0.0 is in GAC.

2. Create an xunit test project with dotnet new xunit.

3. In the generated .csproj file, change the target framework to net46. Also change the package reference versions.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net46</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
  </ItemGroup>

</Project>

1. Open the project in Visual Studio, rebuild the project, then look at the Test output pane

Expected:

No warnings or errors

Actual:

there’s a warning

[xUnit.net 00:00:00.0226725] Skipping: test4 (could not find dependent assembly 'Microsoft.Extensions.DependencyModel, Version=1.1.0')

1. Right click the test in Test Explorer then select Run Selected Tests

Expected:

Test runs and passes.

Actual:

------ Run test started ------
[xUnit.net 00:00:00.0296561] test4: Catastrophic failure: System.TypeInitializationException: The type initializer for 'Xunit.DiaSession' threw an exception. ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Reflection.TypeExtensions, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
   at Xunit.DiaSession..cctor()
   --- End of inner exception stack trace ---
   at Xunit.DiaSession..ctor(String assemblyFileName)
   at Xunit.DiaSessionWrapper..ctor(String assemblyFilename)
   at Xunit.XunitFrontController..ctor(AppDomainSupport appDomainSupport, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, ISourceInformationProvider sourceInformationProvider, IMessageSink diagnosticMessageSink)
   at Xunit.Runner.VisualStudio.TestAdapter.VsTestRunner.RunTestsInAssembly(IRunContext runContext, IFrameworkHandle frameworkHandle, LoggerHelper logger, IMessageSinkWithTypes reporterMessageHandler, AssemblyRunInfo runInfo)
========== Run test finished: 0 run (0:00:00.427418) ==========

Environment

Operating system: Windows 10: a Microsoft internal selfhost version)

vstest.executionengine.x86.exe: 15.00.26228.0

My findings

• I am using the SDK-style .csproj to target net46. According to the doc, Desktop, UWP & Native unit testing continues to use the test platform (TPv1) located @ “%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow”.. (I have briefly tried the method of putting a testplatform.config mentioned at the same link but it didn’t work for me.)

• I have .NET 4.7 on my repro machine. I suspect that System.Runtime, Version=4.1.0.0 is in the GAC for latest windows builds.

• When discovering/executing unit tests, VS tries to get additional test extensions from both xunit assemblies in the following order

  • %TEMP%\VisualStudioTestExplorerExtensions\2.2.0\build\netcoreapp1.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll
  • %Temp%\VisualStudioTestExplorerExtensions\2.2.0\build\_common\xunit.runner.visualstudio.testadapter.dll

• Because of the GAC version of System.Runtime.dll, Version=4.1.0.0, xunit test extension from the first assembly (.NET Core version) is successfully added.

• The test extension from the second assembly (Desktop version) is also added.

• Now VS test platform has two test adapters for executor://xunit/VsTestRunner2

• When I rebuild the solution, the VS Test discovery engine uses the first one (.NET Core version) and failed to load its dependency Microsoft.Extensions.DependencyModel, Version=1.1.0.0 because my test is targeting net46. Note, I am not clear why this dependency didn’t cause problem when VS tries to get types via reflection from assembly xunit.runner.visualstudio.dotnetcore.testadapter.dll. It seems still able to find the test (using the second xunit test adapter?)

------ Discover test started ------
[xUnit.net 00:00:00.0042451] Skipping: test4 (could not find dependent assembly 'Microsoft.Extensions.DependencyModel, Version=1.1.0')
[xUnit.net 00:00:00.2620017]   Discovering: test4
[xUnit.net 00:00:00.3980642]   Discovered:  test4
========== Discover test finished: 1 found (0:00:00.6219404) ==========

• When I run a test from VS Test Explorer, it failed.

• On my non-reproing machine that have .NET 4.6. VS also tries to get test extensions from the same two xunit assemblies. But getting types via reflection fails for the .NET Core version because of the missing dependency on System.Runtime, Version=4.1.0.0. The ReflectionTypeLoadException is handled. Thus only the Desktop version of xunit test adapter is added to the list of test discoverers.