vstest: dotnet test does not forward MSBuild properties to msbuild in .NET 7 RC1

Description

Starting with .NET SDK 7.0.100-rc.1.22431.12 MSBuild parameters specified on the command line to dotnet test are no longer forwarded to MSBuild.

The issue was not present in .NET 7 preview 7.

Steps to reproduce

Clone the martincostello/api repository at commit 61ec79c1c1e773b721f6ebdf28ed189d78d63ae7 and then run dotnet test with the arguments specified below.

git clone https://github.com/martincostello/api
cd api
git checkout 61ec79c1c1e773b721f6ebdf28ed189d78d63ae7
dotnet test ./tests/API.Tests --configuration Release --filter Category=EndToEnd /p:CollectCoverage=false

Expected behavior

dotnet test exits with an exit code of 0.

Actual behavior

dotnet test returns a non-zero exit code due to the CollectCoverage property not being set to false, causing the minimum coverage requirement of coverlet to not be satisfied.

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+
| API    | 0%   | 0%     | 0%     |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

C:\Users\martin.costello\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error : The minimum line coverage is below the specified 55 [C:\Coding\martincostello\api\tests\API.Tests\API.Tests.csproj] C:\Users\martin.costello\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error : The minimum branch coverage is below the specified 55 [C:\Coding\martincostello\api\tests\API.Tests\API.Tests.csproj]
C:\Users\martin.costello\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error : The minimum method coverage is below the specified 55 [C:\Coding\martincostello\api\tests\API.Tests\API.Tests.csproj]
C:\Users\martin.costello\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error :  [C:\Coding\martincostello\api\tests\API.Tests\API.Tests.csproj]
C:\Users\martin.costello\.nuget\packages\coverlet.msbuild\3.1.2\build\coverlet.msbuild.targets(71,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in /_/src/coverlet.msbuild.tasks/CoverageResultTask.cs:line 247 [C:\Coding\martincostello\api\tests\API.Tests\API.Tests.csproj]

Environment

❯ dotnet --info
.NET SDK:
 Version:   7.0.100-rc.1.22431.12
 Commit:    f1cf61e1c0

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22000
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.100-rc.1.22431.12\

Host:
  Version:      7.0.0-rc.1.22426.10
  Architecture: x64
  Commit:       06aceb7015

.NET SDKs installed:
  2.1.818 [C:\Program Files\dotnet\sdk]
  3.1.120 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.302 [C:\Program Files\dotnet\sdk]
  3.1.303 [C:\Program Files\dotnet\sdk]
  3.1.403 [C:\Program Files\dotnet\sdk]
  3.1.423 [C:\Program Files\dotnet\sdk]
  5.0.103 [C:\Program Files\dotnet\sdk]
  5.0.104 [C:\Program Files\dotnet\sdk]
  5.0.203 [C:\Program Files\dotnet\sdk]
  5.0.214 [C:\Program Files\dotnet\sdk]
  5.0.303 [C:\Program Files\dotnet\sdk]
  5.0.408 [C:\Program Files\dotnet\sdk]
  5.0.411 [C:\Program Files\dotnet\sdk]
  6.0.100-rc.1.21458.32 [C:\Program Files\dotnet\sdk]
  6.0.102 [C:\Program Files\dotnet\sdk]
  6.0.109 [C:\Program Files\dotnet\sdk]
  6.0.202 [C:\Program Files\dotnet\sdk]
  6.0.203 [C:\Program Files\dotnet\sdk]
  6.0.304 [C:\Program Files\dotnet\sdk]
  6.0.400 [C:\Program Files\dotnet\sdk]
  6.0.401 [C:\Program Files\dotnet\sdk]
  7.0.100-rc.1.22431.12 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0-preview.8.20414.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.0-preview.7.22376.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.0-rc.1.22427.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0-preview.8.20407.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-preview.5.21301.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-rc.1.21451.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-rc.2.21480.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.0-preview.7.22375.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.0-rc.1.22426.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.28 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0-preview.8.20411.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0-preview.5.21301.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0-rc.1.21451.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0-rc.2.21501.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.0-preview.7.22377.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.0-rc.1.22427.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  arm64 [C:\Program Files\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  C:\Coding\martincostello\api\global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

AB#1663078

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 17
  • Comments: 34 (12 by maintainers)

Commits related to this issue

Most upvoted comments

7.0.101 will be the first monthly servicing release of the 7.0.1xx SDK feature band, so it will arrive early/mid next month around the same time as all of our monthly servicing releases.

@baronfel I’d like to leave the feedback that this is one of those large issues where the fix shouldn’t be delayed by over a month to coincide with fixed release schedules. This should have been fixed as fast as possible. Especially seeing all the comments here about people getting this issue pushed on them via updates of their github actions etc.

Hello folks! A quick update to let you know that the fix was just merged and will be available in 7.0.101. Once again really sorry in the delay in fixing the issue!

.NET SDK 7.0.101 was released today and I can confirm it fixed problems on our build server.

Hello folks! A quick update to let you know that the fix was just merged and will be available in 7.0.101. Once again really sorry in the delay in fixing the issue!

Thanks for getting the fix in. What surprises me is that a regression like this landed in the release in the first place. Especially since coverlet appears to be the de facto standard coverage provider and their official tutorial makes heavy use of this mechanic this should have been a release blocker, is there any reason why it wasn’t?

7.0.101 will be the first monthly servicing release of the 7.0.1xx SDK feature band, so it will arrive early/mid next month around the same time as all of our monthly servicing releases.

Hey there! Sorry for the long delay, we have an open PR with the fix that is waiting for validation and we will bring the subject to the table so we can push the release as fast as possible. Will ping back here as soon as it’s available.

As a workaround, since MSBuild accepts environment variables as property values you could set an env var for each /p property you needed for the duration of the command.

Good spot, thanks for the ping @slang25. We’ll take a look and get this fixed for rc2.

FYI It seems to accepts the first msbuild property but nothing after. This is causing issues throughout all our pipelines! While the environment variable solution would work, we have to update all our shared and hardened AZDO templates (likely to revert once this is fixed) which we aren’t crazy about.

This should’ve been corrected for GA…

Yeah, I know it sucks, but a workaround (as mentioned by @baronfel) is to add the parameters as environment variables in your build pipeline; here is an Azure example from my own project:

  - task: DotNetCoreCLI@2
    displayName: 'Test Solution - Linux'
    condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
    inputs:
      command: test
      projects: test/**/*.csproj
      arguments: '--configuration $(BuildConfiguration) --verbosity quiet --collect:"XPlat Code Coverage" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover'
      publishTestResults: true
    env:
      CollectCoverage: true
      CoverletOutputFormat: opencover

If you want more context, have a look here: https://github.com/gimlichael/Cuemon/blob/development/.azure/steps/unit-test.yml

Is there an update on whether this issue will be fixed into time for .NET 7 GA?

Nope, the PR has not been merged 😢

Just to add to your voice: we see this issue too when kicking off tests from this PS task.

task RunTests -If {!$SkipTests -and $SolutionToBuild} {
    exec { 
        dotnet test $SolutionToBuild `
                    --configuration $Configuration `
                    --no-build `
                    --no-restore `
                    /p:CollectCoverage="$EnableCoverage" `
                    /p:CoverletOutputFormat=cobertura `
                    /p:ExcludeByFile="$($ExcludeFilesFromCodeCoverage.Replace(",","%2C"))" `
                    --verbosity $LogLevel `
                    --logger $DotNetTestLogger
    }
}

Thanks for the update @matyasbach. I will move forward by closing this issue as the fix is now released.

@heaths The work around is to not use .net 7.0.100, because that is where the bug originates.

This also breaks SqlClient tests. Is there an ETA on the fix?

Another option is to add the necessary properties to csproj/Directory.Build.props, e. g.:

<PropertyGroup Condition="$(MSBuildProjectName.Contains('.Tests.'))">
  <CollectCoverage>true</CollectCoverage>
  <ExcludeByAttribute>GeneratedCodeAttribute</ExcludeByAttribute>
  <Exclude>[*]*.Migrations.*</Exclude>
  <ExcludeByFile>**/MyFile.cs</ExcludeByFile>
  <CoverletOutputFormat>opencover</CoverletOutputFormat>
</PropertyGroup>

This also breaks SqlClient tests. Is there an ETA on the fix?

Hi @JRahnama, the fix was done, we are just waiting for the releasing. For the ETA please see https://github.com/microsoft/vstest/issues/4014#issuecomment-1311751668

Hi, for the people could not wait for release 7.0.101. I was able to build the fixed version of dotnet.dll. You can download from: Here Just copy it in C:\Program Files\dotnet\sdk\7.0.100 and now it will work.