vstest: 'dotnet test' in solution folder fails when non-test projects are in the solution

The dotnet pack command looks for the <IsPackable> property. It would be nice if dotnet test looked for <IsTestable>. At face value, it seems like a really simple solution that would be quite valuable.

fyi I wrote a blog post about my solution (enables dotnet test without needing to implicitly build projects multiple times) https://dasmulli.blog/2018/01/20/make-dotnet-test-work-on-solution-files/

I’m currently using these msbuild targets in the solution directory to work around this:

A Directory.Build.targets file makes sure that all projects have a VSTestIfTTestProject target:

<Project>
  <Target Name="VSTestIfTestProject">
    <CallTarget Targets="VSTest" Condition="'$(IsTestProject)' == 'true'" />
  </Target>
</Project>

Then file named after.MySolutionName.sln.targets adds a VSTest target to the solution itself that dotnet test will call:

<Project>
  <Target Name="VSTest">
    <MSBuild Projects="@(ProjectReference)" Targets="VSTestIfTestProject" />
  </Target>
</Project>

+1 to the requirement of ‘works out of the box’.

@livarcocc I ran into the same issue. There’s a quite good workaround for that: create a Directory.Build.targets in the root of your repository with the following content:

<Project>
  <!--Override VSTest target when the project is no test project.-->
  <Import Project="OverrideTest.targets" Condition="'$(IsTestProject)' != 'true'"/>
</Project>

create the file OverrideTest.targets beside the Directory.Build.targets

<Project>
  <Target Name="VSTest">
  </Target>
</Project>

what happens here is that the VSTest target is overridden as empty (nothing happens) when the project is not considered as test project by msbuild. A better solution (reusable across many repositories) is to put that into an build nuget package and just reference that package in your Directory.Build.targets, but unfortunately I can’t publish mine on nuget.org

@consultwithmike I don’t think it’s reasonable to expect every developer or organization to restructure all their solutions to work around this. Only testing projects that have some test sdk as a dependency - or supporting some project name filter - seems to me like a much more reasonable expectation.

@smadala Please pick this up for resolution as discussed.

While the current solution for @consultwithmike works, it might be convenient to run all your tests from a single command, to ease stat parsing. This can be achieved with dotnet vstest. However, you are required to specify every dll separately.

For my project, this results in the following command (I know, the folders and casing are wrong, which is what you get when you maintain a .NET project originally created by Java developers):

dotnet vstest modules/**.tests/bin/Release/**/**.tests.dll

It wil run all test in the dll’s matching the argument selector, and give some final, accumulated output:

Total tests: 799. Passed: 797. Failed: 0. Skipped: 2.
Test Run Successful.
Test execution time: 50.5416 Seconds

This method does not appear to be working from PowerShell.

The workaround I’m using right now is this script.

.\GenerateTestSolution.ps1 .\Foo.sln

It simply strips out projects that don’t end with .Tests from a new copy of the solution file. To run the tests, do something along the lines of:

test Foo.Tests.sln -c %CONFIGURATION% --no-build

--no-build prevents build failures due to missing projects.

HTH

In powershell I am using this:

 dotnet vstest ((ls -Recurse *.Test.Unit.dll | % FullName) -Match "\\bin\\Release\\")

Meaning “run all tests recursively in all sub-folders, where assemblies are named *.Test.Unit.dll and their paths contain \bin\Release\”.

Pretty sure it could be improved (by using proper directory filtering instead of string match, for instance), but it gets the job done.

I noticed that using dotnet 2.1.500 doing this

dotnet test MySolution.sln

provides information that tests are skipped if they not include IsTestProject = true

Skipping running test for project xxx\src\xxx\MyProject.csproj. To run tests with dotnet test add "<IsTestProject>true<IsTestProject>" property to project file.

Was the issue already fixed with a release? I stumbled on this issue in a Build Pipeline on Azure DevOps using Linux Ubuntu 1604 host, running a .Net Core task

Task         : .NET Core
Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet.
Version      : 2.141.5
Author       : Microsoft Corporation
Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=832194)

but the same was not happening using a Linux Preview Host, reporting the same task information.

I’d still prefer a solution that would treat it as a single test run. all solutions involve invoking vstest multiple times, either through sh commands or looking at all projects with $(IsTestProject) in the solution (see my blog post). But none them can:

• Synchronize access to shared resources implemented through constructs like fixtures implemented by the testing framework or manually (un-)locking of shared resources.
• Generate a single run result filej

UPDATE

Even executing that command wasn’t working right so I had to build this bash script to do the job. Now I just execute the bash script.

#!/bin/bash

clear
for i in Tests/* ; do
  if [[ -d "$i" && "$i" == *.Tests ]]; then
    limit=$((${#i}+20))

    echo "Executing Tests for $i"
    printf '=%.0s' $(seq 1 $limit)
    echo ""

    dotnet test "$i" --verbosity quiet --no-build

    echo ""
  fi
done

The best way to solve this is to place all of your Tests project into their own folder, which is relatively good practice to keep large projects manageable. So, let’s assume a structure like this:

- SolutionFolder
  - Project1
  - Project2
  - Tests
    - Project1.Tests
    - Project2.Tests

Now to run all tests you just use dotnet test Tests/**.

It would be great if this would work out of the work!

@consultwithmike I’m glad you found a solution that works for you. I’m just making the point that the work-arounds are not viable options for everyone and that this is still an active issue for many of us.

To me it is working as expected on 2.1.500! It looks for projects with <IsTestProject>true</IsTestProject> in the solution.

Finally, after more than 1 year waiting it works without workarounds! 🙂

For people who can’t make it work with Travis:

It started returning code 0 when I changed my travis.yml file from

language: csharp
solution: MySolution.sln
mono: none
dotnet: 2.1.402
script:
- dotnet test TestFolder/testProj.csproj

to

language: csharp
solution: MySolution.sln
mono: none
dotnet: 2.1.402
script:
- cd TestFolder
- dotnet test

I know this could be a strange solution but I personally do the following.

1. Create a separate solution configuration (I usually name it “Test”) using Visual Studio Configuration Manager (do not create project configurations, select to copy configuration from the configuration you use for build i.e. “Release”).
2. Now set build checkboxes only for Test projects.
3. Execute dotnet test -c Test --no-build --no-restore on the solution

The CLI would run tests only for test projects.

Now when new Test project is added only a solution configuration has to be modified (no CI changes involved).

Hope this helps.

@dasMulli Your method works, but it does result in several invocations of the test runner, leading to several output files and a longer execution time. It also does not generate an overview of all tests run in the end.

Due to this, I personally prefer the dotnet vstest solution as it results in a single output file, which runs faster and generates an overview at the end of execution.

@jcdickinson That script worked really well. I was able to get my build process to run the tests without failing. Thanks!