azure-pipelines-tasks: VsTest Code Coverage is broken by design
Environment
- Azure DevOps Server 2019 Update 1 Patch 1 (Dev17.M153.3)
- Agent v2.153.2 inside Docker Container based on
mcr.microsoft.com/dotnet/framework/runtime:4.8-20190709-windowsservercore-1903 - Microsoft.TestPlatform v16.3.0 installed via
VsTestPlatformToolInstaller
Issue Description
With codeCoverageEnabled: true the VsTest task generates a proprietary .coverage file which cannot be consumed by Azure DevOps / PublishCodeCoverageResults nor is it published as an artifact.
The only supported formats are Cobertura and JaCoCo.
Reading the docs for PublishCodeCoverageResults it says :
Tasks such as Visual Studio Test, .NET Core, Ant, Maven, Gulp, Grunt also provide the option to publish code coverage data to the pipeline. If you are using these tasks, you do not need a separate Publish Code Coverage Results task in the pipeline.
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results
This is wrong because the VsTest task neither converts the resulting .coverage to a supported file format nor uploads the raw .coverage file as an artifact.
But the story continues…
So if the task itself does not produce a valid output why not simply convert the .coverage file to Cobertura using danielpalme/ReportGenerator and then use PublishCodeCoverageResults ?
Unfortunately this is also not possible because of #6279.
The .coverage file does not exist anymore after a test run so there is nothing we could do with it.
This effectively renders the complete code coverage feature of the VsTest task useless.
Obviously we could use a simple script to execute our tests with enabled code coverage and then do the file conversion, but then we would also loos all the nice features of the VsTest task like distributed batching and flaky test detection.
Proposal
It would be great to get some generic infrastructure to use any code coverage tool which would resolve the issue in the first place. Because most of the the time these tools works the same (you just run them with an argument telling them what other executable to run) this should be quite simple?
Degenerated example using dotCover:
dotCover.exe cover --TargetExecutable=$(vsTestExecutable) --TargetArguments=$(vsTestArguments) --ReturnTargetExitCode
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 10
- Comments: 18 (8 by maintainers)
@ShreyasRmsft @anshii03 this issue needs to be definitely re-opened, we are facing a similar problem (very well described here https://github.com/microsoft/vstest/issues/981#issuecomment-1034093540)
This issue needs to be Re - Opened because the proposed feature enable to display the html by default never came to light. And every new user of VsTest task integrating on the ADO is facing the same race condition.
Proposal: why can’t codecoverage (from VsTest task) also output the .xml by default? It’s upto the use to consume it or not.
Is this code coverage view going to be made available soon for Azure DevOps Server 2019? It appears to only be available currently for microsoft hsoted Azure DevOps services.
I’m interested and emailed devops_tools@microsoft.com, but received this in response:
VsTest Task do upload .coverage files as an artifact.
And Publish Code Coverage Task don’t publish multiple code coverage reports as mentioned in doc https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops#q--a
Alternatively , Private preview is ready for the new first class view :
If anyone is interested , please mail your account details on devops_tools@microsoft.com. We would love to have your feedback before it becomes GA.
Hence , closing this ticket.