azure-pipelines-tasks: DotNetCoreCLI@2 reports wrong number of tests
I am using yaml pipeline to run unit tests. As unit testframework we are using xUnit. In our solution we have 3 test projects with tests. .NET Core 3.1 is in use.
We configured the pipeline like this to run the tests:
steps:
- task: DotNetCoreCLI@2
displayName: 'DotNetCoreCLI: Run UnitTests'
inputs:
command: test
projects: ${{parameters.TestProjects}}
arguments: '--configuration ${{parameters.BuildConfiguration}} --collect "Code coverage"'
Resulting command per test project:
[command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\<TestProjectFolder>\<TestProject>.csproj --logger trx --results-directory
All tests from all test projects are executed. See output from the test projects:
TestProject-1)
2020-11-05T12:54:34.9667677Z D:\a\_temp\369999cb-39f5-48f0-a7f8-403f53016a10\VssAdministrator_WIN-TOJS93SFU8U_2020-11-05.12_54_26.coverage
2020-11-05T12:54:34.9688121Z Test Run Successful.
2020-11-05T12:54:34.9688521Z Total tests: 31
2020-11-05T12:54:34.9696009Z Passed: 31
2020-11-05T12:54:34.9696824Z Total time: 7.9326 Seconds
TestProject-2)
2020-11-05T12:55:20.8713307Z D:\a\_temp\a133b198-82c0-48c2-bfda-c86e1c3dd79b\VssAdministrator_WIN-TOJS93SFU8U_2020-11-05.12_55_09.coverage
2020-11-05T12:55:20.8714296Z Test Run Successful.
2020-11-05T12:55:20.8714624Z Total tests: 470
2020-11-05T12:55:20.8715010Z Passed: 470
2020-11-05T12:55:20.8715362Z Total time: 9.9378 Seconds
TestProject-3)
2020-11-05T12:55:41.9953337Z D:\a\_temp\5a1cc4e0-1fe9-480b-82fc-6ce4b1ab153f\VssAdministrator_WIN-TOJS93SFU8U_2020-11-05.12_55_35.coverage
2020-11-05T12:55:41.9978867Z Test Run Successful.
2020-11-05T12:55:41.9986681Z Total tests: 104
2020-11-05T12:55:41.9988231Z Passed: 104
In total there are 605 tests. Can confirm that with VS and R# testrunner. That is the expected number.
Issue: In the build pipline at tab “test” there are just 248 total tests reported which I don’t understand. The interesting part from the build log file might be the publishing part:
2020-11-05T12:55:43.4954681Z Result Attachments will be stored in LogStore
2020-11-05T12:55:43.5510451Z Run Attachments will be stored in LogStore
2020-11-05T12:55:43.7578986Z Info: Azure Pipelines hosted agents have been updated to contain .Net Core 3.x (3.1) SDK/Runtime along with 2.1. Unless you have locked down a SDK version for your project(s), 3.x SDK might be picked up which might have breaking behavior as compared to previous versions.
2020-11-05T12:55:43.7580331Z Some commonly encountered changes are:
2020-11-05T12:55:43.7581897Z If you're using `Publish` command with -o or --Output argument, you will see that the output folder is now being created at root directory rather than Project File's directory. To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2020-11-05T12:55:43.7587697Z ##[section]Async Command Start: Publish test results
2020-11-05T12:55:44.0491621Z Publishing test results to test run '18550'.
2020-11-05T12:55:44.0575929Z TestResults To Publish 175, Test run id:18550
2020-11-05T12:55:44.0604256Z Publishing test results to test run '18552'.
2020-11-05T12:55:44.0606072Z TestResults To Publish 47, Test run id:18552
2020-11-05T12:55:44.0640259Z Publishing test results to test run '18548'.
2020-11-05T12:55:44.0642994Z TestResults To Publish 26, Test run id:18548
2020-11-05T12:55:44.0665928Z Test results publishing 26, remaining: 0. Test run id: 18548
2020-11-05T12:55:44.0668711Z Test results publishing 47, remaining: 0. Test run id: 18552
2020-11-05T12:55:44.0672538Z Test results publishing 175, remaining: 0. Test run id: 18550
It just publish 248 tests (175, 47, 26). But why?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17
It does surprise me as well, especially given the popularity of XUnit.
If the table view below understands the distinction between
[Fact]and[Theory]tests then their data structure already supports it, so it’d just be a matter of changing the view’s query to update the total.Same deal here. Wish someone would give this issue some love.
I don’t understand why this issue was not fixed. It is totally confusing.
SUMMARY
DEVS - please respond in some fashion rather than ignoring this. It is causing us significant issue as we have many Theory type tests and literally have a tiny number of tests reported from Azure even though we have over 1600 tests via Theory data.
Also can confirm that this is happening in our .NET Core xUnit project. The unit test count shows correctly on the .trx file but the published results is wrong
Unit test task
Publish test task
This is what our .yml pipeline looks like:
Is there any solution to this issue?
Can confirm this is happening with Xunit. Searching for how to resolve this. After setting up the pipeline and running it, on the Tests tab, the number shown reflects the number of methods in the Xunit test project that was run. Then there is a drop down to click, and there its shown all the inline data that ran. These inline data tests are not reflected in the total test count. Also using .NET Core 3.1
Hi @davidfordaus! I am the author of this issue. I created it on November 2020. From my point of view this is definitely a bug. I also discussed it with my teammates and they agree on it. I also asked here the audience for some feedback or consideration if this is by design and if yes what is the idea behind and why to distinguish between a simple test method (a single test) and parameterised tests (with the Theory attribute and inline data or from a data source).
However, I am never able to explain the final total number of tests to the team because to be honest at that point we don’t care how the test were written (with or without parameterised tests).
My personal conclusion here is that this is a low priority issue.
Now raised with this bug: https://github.com/microsoft/azure-pipelines-tasks/issues/15030