vstest: 17.4.0 Breaks pipeline code coverage

Description

We’re using Azure YML pipelines, and running the dotnet CLI to build and run tests since we’re using Linux as the build agent. Our test step has the following parameters:

    - task: DotNetCoreCLI@2
      displayName: Run Unit Tests
      inputs:
        command: 'test'
        projects: '$(testProjects)'
        arguments: '--configuration $(BuildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx'

And this used to work - we used to get code coverage results during our dotnet test step, the logs would show:

Calculating coverage result...	
  Generating report '/home/vsts/work/1/s/Data API/Tests/DataApi.API.Tests/coverage.opencover.xml'	
+------------------------+------+--------+--------+	
| Module                 | Line | Branch | Method |	
+------------------------+------+--------+--------+	
| DataApi.Infrastructure | 0%   | 0%     | 0%     |	
+------------------------+------+--------+--------+	
| DataApi.API            | 5.1% | 2.17%  | 11.65% |	
+------------------------+------+--------+--------+	
| DataApi.DomainModel    | 0%   | 0%     | 0%     |	
+------------------------+------+--------+--------+	
+---------+-------+--------+--------+	
|         | Line  | Branch | Method |	
+---------+-------+--------+--------+	
| Total   | 2.34% | 1.08%  | 6.15%  |	
+---------+-------+--------+--------+	
| Average | 1.7%  | 0.72%  | 3.88%  |	
+---------+-------+--------+--------+

Now we tried to run another build recently, but it failed to pick up any code coverage on the build whatsoever. Our Azure pipeline YML file is exactly the same, yet the logs are slightly different.

The main difference being, that the code coverage graphic I showed above simply doesn’t exist in the logs - that step seems to be completely missed. The only thing I can see that’s changed in the log is from this version (where it worked): Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64) to this version (where it no longer works): Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)

Nothing else of note has changed in the build logs apart from that - and now we are unable to get code coverage. Has something in the interface changed that means I can’t use the same command? Or is there a bug? What’s going on here?

We’re using ubuntu-latest on Azure cloud hosted agents.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 27 (13 by maintainers)

Most upvoted comments

@MarcoRossignoli , For me the problem was #4014 and I get it working by adding the params to env variables

    inputs:
      command: test
      projects: test/**/*.csproj
      arguments: '--configuration $(BuildConfiguration)  /p:CollectCoverage=true /p:CoverletOutputFormat=opencover'
    env:
      CollectCoverage: true
      CoverletOutputFormat: opencover

So now I get the same result and it works in SonarCube again.

To sum up, if I understand correctly,

the error is that dotnet test parameters are not forwarded (#4014) in the recently released dotnet 7 which recently became default in azure pipelines agents.
Easy workaround is to use env variables, as I describe above
A better workaround may be to switch to Coverlet collector instead of Coverlet msbuild, since thats not affected by this bug, and is the preferred Coverlet (I didn’t chooose this, cause it was slower (i think --no-rebuild was ignored) and Sonar was not able to find them)
Force another dotnet or vstest version. (I didn’t look into this, but see #4014 )

The real fix will be with the December patch to dotnet7 and that is enabled in azure pipeline.

claus18 on Dec 5, 2022

@aidanhorton can I close this one? Looks like you’ve solved.

@304NotModified @riezebosch @mensurkurtagic can you please open another issue if you’re blocked with your one(they look unrelated to the Aidan one)?

@claus18 is it working for you?

MarcoRossignoli on Dec 5, 2022

@riezebosch sorry I forgot mention 1 more thing. dotnet-coverage tool supports only static instrumentation of macOS arm64. So you need to specify also which files should be instrumented using: -if|--include-files <include-files>

Microsoft.CodeCoverage is instrumenting all files in folder where there is test assembly. To change it you need to add runsettings and specify:

<ModulePaths>
            <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
            </Exclude>
</ModulePaths>

For more info check: https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022

jakubch1 on Dec 5, 2022

macOS arm64 (M1) you need to use this version: https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.5.0-preview-20221205-01 It should reach nuget.org around 28th December.

Another way you could use is https://aka.ms/dotnet-coverage. Latest 17.5.0 version supports macOS arm64 (M1). In this case you can run: dotnet-coverage collect -f cobertura "dotnet test"

jakubch1 on Dec 5, 2022

@aidanhorton nowadays we ship built-in support for coverage using the command line dotnet test --collect "Code Coverage;Format=cobertura" if you’ve issue with Coverlet you can try this one too. You can customize the coverage following the instruction https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test (--collect command line switch).

Thanks for the suggestion! We’re using 2 things that unfortunately defeat that - 1 is we’re using Linux build agents, meaning we have to use XPlat code coverage (which isn’t an issue as that does Cobertura too), but we are also using SonarCloud to analyze our code and our coverage output - and that doesn’t accept Cobertura format - only opencover and a few others I believe.

Either way, I’ve managed to get it working with the opencover changes I mentioned in an earlier message - but sounds like others are having a few problems.

aidanhorton on Dec 5, 2022

I don’t know if a certain style of arguments got made obsolete in either the dotnet CLI or the test execution CLT, but if I change the arguments of the dotnet test command as follows:

Then it fixes the issue.

aidanhorton on Nov 28, 2022