runtime: `dotnet test` is not running tests for .NETFramework targets

Description

When running dotnet test on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn’t find an active one.

Reproduction Steps

build clr+libs -rc Release cd src\libraries\System.Collections.Immutable\tests dotnet test -f:net462

Expected behavior

Tests run.

    Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod,
   method display options = None)
    Discovered:  System.Collections.Immutable.Tests (found 4008 of 4123 test cases)
    Starting:    System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16)

Actual behavior

No tests run:

Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Regression?

I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test.

Known Workarounds

Directly invoke the test target.

\src\dotnet\runtime.dotnet\dotnet build /t:test -f:net462

Configuration

No response

Other information

No response

About this issue

Original URL
State: open
Created 8 months ago
Comments: 15 (8 by maintainers)

Most upvoted comments

you should stop doing that. It’s unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages or some equivalent).

Just to share some context. We invoke our tests in CI on different machines than the build agents. We use Helix for that, which is a distributed machine testing orchestration infrastructure. Because of that, we can’t rely on the package cache. Instead we should have just sent that dependency as an additional “payload” (dependency) to that helix test agent.

Unfortunately our infrastructure here has become quite complex with so many different app models and ways of testing that this change will probably require some time.

But for what is worth, this is indeed our own fault. Thanks for help diagnosing the issue 😃

ViktorHofer on Oct 30, 2023

Yeah, that operation makes no sense to me. You were basically just waiting to explode. 😁

Each runner has to bring along their own copy of xunit.runner.utility.* that they’re linked against. It’s just the VSTest adapter which insists on copying it into your output folder; all the other runners just live elsewhere and are invoked by hand, so the assemblies are never mixed together.

bradwilson on Oct 30, 2023

For a bit of architectural background…

xUnit.net splits what it does into two separate sections which do not directly interact except via very strict contracts. There are “runner” and “execution” halves of the system. In v1/v2 when using .NET Framework, these two halves can be optionally separated by an app domain boundary (and in v2 .NET Core, they live in the same process). In v3 against all target platforms, they’re separated by a process boundary.

Tests lives in the “execution” half, and are linked against xunit.core and xunit.execution.* (and usually xunit.assert, though that’s optional).
Runners live in the “runner” half, and are linked against xunit.runner.utility.* (and maybe other runner related things).
The only common assembly that lives in both halves, and comprises the communication contract, is xunit.abstractions. It has been frozen since we shipped 2.0 (we have shipped newer versions of the NuGet assembly just to add extra targets as 2.0 was back in PCL259 days, but the DLL itself remains unchanged since we shipped it).

By keeping this strong separation and frozen contracts, we have both backward compatibility (newer runners can run older tests) and major forward compatibility (older v2 runners can run newer v2 tests). In point of fact, I just was able to successfully run tests linked against core framework 2.5.3 with the VSTest adapter 2.0.0, which is 8.5 years old and targeted net20, wpa81, and win8. Obviously .NET Core didn’t even exist back then, so it wouldn’t be able to run a platform it doesn’t know about, but .NET Framework is not a problem.

If your tests link against xunit.runner.utility.*, you’re bringing the runner half of things where it normally doesn’t live or belong. This is unavoidable if you’re writing your own runner, and want to test that runner (we obviously do this, but it should be very very rare). Normally this wouldn’t be a problem except when you’re using VSTest, because there is a requirement placed by VSTest that we copy the VSTest adapter into your output folder, which of course means we also have to copy the runner utility libraries (xunit.runner.utility.* and xunit.runner.reporters.*). This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you’re in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you’re using.

In addition, there is no guarantee that any particular version of xunit.runner.visualstudio will use any particular version of xunit.runner.utility. We shipped 2.5.3 of the former at the same time as 2.5.2 of the latter, but even simultaneous shipment isn’t guaranteed (and in fact they lived very separate lives back when @clairernovotny owned the VSTest adapter).

When I shipped core framework 2.5.3, it had no runner utility changes that would’ve necessitated bringing in a new version for the VSTest adapter, so I didn’t bother to ship a new one. It appears that when I ship the next core framework drop (which I’m planning to number 2.6.0), this will still be true, so again I’m not likely to ship another VSTest adapter version.

bradwilson on Oct 30, 2023