Duly noted. Believe me, I am not happy about this either. We are looking into conditionally reverting this change until it is fixed in azdo.

This is not a bug, mstest now reports data driven tests as single tests. This was done to bring parity to how xunit and nunit does it.

@Haplois There are few duplicates of this issue. Could you link them all to a single one, and post an update on when the work on AzDO will be finished, please?

Also correct my statement above if I am wrong 😃

@nohwnd
The ONLY reason I and many other engineers use MSTest is because of the seamless integration with Azure DevOps Test Hub. If you cannot maintain this integration you are just undermining the project.

Here are my concerns:

1. The team broke this integration a year ago and it has yet to be addressed
2. The interesting (wrong) assertion that this is “not a bug”
3. The trx report format has fundamentally changed and that doesn’t seem to justify a major or minor version bump
4. I know this project and the Azure DevOps Visual Studio Test task are separate projects, but this kind of thing has the stink of the bad old days of Microsoft. - AKA Microsoft cannot seem to even work with itself

As far as I can tell here are the related/duplicate issues:

• https://github.com/microsoft/testfx/issues/860
• https://github.com/microsoft/testfx/issues/955
• https://github.com/microsoft/testfx/issues/992
• https://github.com/microsoft/testfx/issues/1023
• https://github.com/microsoft/testfx/issues/1026
• https://github.com/microsoft/testfx/issues/1035

I forgot to mention that the link you posted above (link) seems to go to an MS internal ADO. Do you have any public information on that issue?

P.S. It also could be that our TFS server is just not configured correctly. I would guess that its configuration has not been revisited since ~2015. It seems like it just doesn’t deliver the source code developed in VS 2022 to the Azure DevOps agent correctly. The easy solution is to move to GitHub. Will try to help the team do it.

Update 11/12/22: I was able to create a completely new folder (not a new repo, but a new folder) in our TFS repo, upload the 2022 version of my test solution with the latest adapters and latest test agent as shown above, and I got good results, with the correct number of test cases displayed in Azure DevOps. This means the older TFS will work as a sources repository, I just had to use a fresh folder. It must have been that there was something being retained in the older folder which contained the VS 2015 version of the test solution. For others still getting errors, I would recommend the same approach: try your old source repository but use a fresh folder and make sure you convert your solution to the latest Visual Studio and the latest adapters.

Just want to mention I am working on making sure I am giving you the correct information. We have some very legacy elements in our system, including the fact that our ADO is hooked up to a Team Foundation server as the source repository. I am making some changes to ensure that I am giving you the right info.

Hey there. Thanks for bearing with us. We did more investigation with @dogirala and here are the results.

Code example

When running the tests for the following code:

namespace MSTestProject
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
        }

        [DataRow(0)]
        [DataRow(1)]
        [DataRow(2)]
        [TestMethod]
        public void DataRowTestMethod(int i)
        {
        }
    }
}

MSTest up to 2.2.3 (included)

Result file (.trx)

The test result file (.trx) would look like this:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="89196f25-1585-46cb-9429-10e74cc201f4" name="currentUser@MyMachine 2022-11-08 15:25:41" runUser="currentUser" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2022-11-08T15:25:41.2734026+01:00" queuing="2022-11-08T15:25:41.2734030+01:00" start="2022-11-08T15:25:39.4674650+01:00" finish="2022-11-08T15:25:41.2866232+01:00" />
  <TestSettings name="default" id="a8af139d-1d5f-4b27-a369-5c0ee9dcc696">
    <Deployment runDeploymentRoot="currentUser_MyMachine_2022-11-08_15_25_41" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="6c135de3-73eb-4cbe-8f39-337340a6d650" testId="12713494-ea68-505c-1ae3-a995253433bd" testName="TestMethod1" computerName="MyMachine" duration="00:00:00.0077905" startTime="2022-11-08T15:25:40.8935789+01:00" endTime="2022-11-08T15:25:40.9730048+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="6c135de3-73eb-4cbe-8f39-337340a6d650" />
    <UnitTestResult executionId="7c368e52-7c43-4b1a-8696-e12d5003d061" testId="cffe872d-a94b-a72a-efd2-1fb9ec28c26b" testName="DataRowTestMethod" computerName="MyMachine" duration="00:00:00.0034511" startTime="2022-11-08T15:25:40.9841452+01:00" endTime="2022-11-08T15:25:40.9899959+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="7c368e52-7c43-4b1a-8696-e12d5003d061" resultType="DataDrivenTest">
      <InnerResults>
        <UnitTestResult executionId="6deb9d3f-02f8-4592-a7ad-da2c307b48d8" parentExecutionId="7c368e52-7c43-4b1a-8696-e12d5003d061" testId="cffe872d-a94b-a72a-efd2-1fb9ec28c26b" testName="DataRowTestMethod (0)" computerName="MyMachine" duration="00:00:00.0001003" startTime="2022-11-08T15:25:40.9841452+01:00" endTime="2022-11-08T15:25:40.9899959+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="6deb9d3f-02f8-4592-a7ad-da2c307b48d8" dataRowInfo="1" resultType="DataDrivenDataRow" />
        <UnitTestResult executionId="1b8b1f06-a581-40ad-b8fd-af918c42b9b8" parentExecutionId="7c368e52-7c43-4b1a-8696-e12d5003d061" testId="cffe872d-a94b-a72a-efd2-1fb9ec28c26b" testName="DataRowTestMethod (1)" computerName="MyMachine" duration="00:00:00.0000113" startTime="2022-11-08T15:25:40.9841452+01:00" endTime="2022-11-08T15:25:40.9899959+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="1b8b1f06-a581-40ad-b8fd-af918c42b9b8" dataRowInfo="2" resultType="DataDrivenDataRow" />
        <UnitTestResult executionId="c9490162-9341-4c9c-9be8-b34528d4b793" parentExecutionId="7c368e52-7c43-4b1a-8696-e12d5003d061" testId="cffe872d-a94b-a72a-efd2-1fb9ec28c26b" testName="DataRowTestMethod (2)" computerName="MyMachine" duration="00:00:00.0000042" startTime="2022-11-08T15:25:40.9841452+01:00" endTime="2022-11-08T15:25:40.9899959+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="c9490162-9341-4c9c-9be8-b34528d4b793" dataRowInfo="3" resultType="DataDrivenDataRow" />
      </InnerResults>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="TestMethod1" storage="c:\src\mstest-playground\mstestproject\mstestproject\bin\debug\net6.0\mstestproject.dll" id="12713494-ea68-505c-1ae3-a995253433bd">
      <Execution id="6c135de3-73eb-4cbe-8f39-337340a6d650" />
      <TestMethod codeBase="C:\src\mstest-playground\MSTestProject\MSTestProject\bin\Debug\net6.0\MSTestProject.dll" adapterTypeName="executor://mstestadapter/v2" className="MSTestProject.UnitTest1" name="TestMethod1" />
    </UnitTest>
    <UnitTest name="DataRowTestMethod" storage="c:\src\mstest-playground\mstestproject\mstestproject\bin\debug\net6.0\mstestproject.dll" id="cffe872d-a94b-a72a-efd2-1fb9ec28c26b">
      <Execution id="7c368e52-7c43-4b1a-8696-e12d5003d061" />
      <TestMethod codeBase="C:\src\mstest-playground\MSTestProject\MSTestProject\bin\Debug\net6.0\MSTestProject.dll" adapterTypeName="executor://mstestadapter/v2" className="MSTestProject.UnitTest1" name="DataRowTestMethod" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="12713494-ea68-505c-1ae3-a995253433bd" executionId="6c135de3-73eb-4cbe-8f39-337340a6d650" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="cffe872d-a94b-a72a-efd2-1fb9ec28c26b" executionId="7c368e52-7c43-4b1a-8696-e12d5003d061" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="5" executed="5" passed="5" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
  </ResultSummary>
</TestRun>

What’s important to note is:

• the total number of tests is 5 and not 4 as we would expect.
• the data row entries are not considered as individual tests but are grouped as InnerResults of the test method (they have the same test ID)
• the TestDefinitions only contains 2 entries

Azure DevOps

Azure DevOps is parsing the trx file and using the results to populate their UI. For the previous trx file, display will be:

• 2/2 reported for the number of tests because they are reading the TestDefinitions entries of the trx.
• The datarow items (DataRowTestMethod (0), DataRowTestMethod (1), DataRowTestMethod (2)) grouped under the one item named DataRowTestMethod because they are using the grouping provided in UnitTestResult of the trx.
• One extra element for TestMethod1

MSTest from 2.2.4 (included)

Result file (.trx)

The test result file (.trx) would look like this:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="f3a5ab2c-96f6-4c7f-a5c7-6f239db27664" name="currentUser@MyMachine 2022-11-08 15:22:22" runUser="currentUser" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2022-11-08T15:22:22.8553679+01:00" queuing="2022-11-08T15:22:22.8553683+01:00" start="2022-11-08T15:22:21.9392535+01:00" finish="2022-11-08T15:22:22.8715965+01:00" />
  <TestSettings name="default" id="d505dfd8-9081-4ad5-b030-af6556d675c2">
    <Deployment runDeploymentRoot="currentUser_MyMachine_2022-11-08_15_22_22" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="61cba81d-ea31-49ba-90e2-392492ded38a" testId="fbcb88c4-070c-4e3c-a6fc-fac91aafc616" testName="DataRowTestMethod (1)" computerName="MyMachine" duration="00:00:00.0000037" startTime="2022-11-08T15:22:22.6686938+01:00" endTime="2022-11-08T15:22:22.6688093+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="61cba81d-ea31-49ba-90e2-392492ded38a" />
    <UnitTestResult executionId="c953f320-0876-425b-9cef-1fa6a3fbfd2f" testId="face297a-edd7-48ce-9aaa-65a72fe2c332" testName="DataRowTestMethod (0)" computerName="MyMachine" duration="00:00:00.0000577" startTime="2022-11-08T15:22:22.6647551+01:00" endTime="2022-11-08T15:22:22.6686331+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="c953f320-0876-425b-9cef-1fa6a3fbfd2f" />
    <UnitTestResult executionId="79730bea-94e1-42e5-81ee-544788341d60" testId="22390aeb-07a5-4cbe-bdcd-2fe6c90b72f7" testName="DataRowTestMethod (2)" computerName="MyMachine" duration="00:00:00.0000034" startTime="2022-11-08T15:22:22.6688280+01:00" endTime="2022-11-08T15:22:22.6690112+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="79730bea-94e1-42e5-81ee-544788341d60" />
    <UnitTestResult executionId="d6e08bfa-b602-497a-8477-3d0711620467" testId="6349b831-af4e-4046-87c2-ec8e3ecffc93" testName="TestMethod1" computerName="MyMachine" duration="00:00:00.0031628" startTime="2022-11-08T15:22:22.6412291+01:00" endTime="2022-11-08T15:22:22.6584147+01:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="d6e08bfa-b602-497a-8477-3d0711620467" />
  </Results>
  <TestDefinitions>
    <UnitTest name="TestMethod1" storage="c:\src\mstest-playground\mstestproject\mstestproject\bin\debug\net6.0\mstestproject.dll" id="6349b831-af4e-4046-87c2-ec8e3ecffc93">
      <Execution id="d6e08bfa-b602-497a-8477-3d0711620467" />
      <TestMethod codeBase="C:\src\mstest-playground\MSTestProject\MSTestProject\bin\Debug\net6.0\MSTestProject.dll" adapterTypeName="executor://mstestadapter/v2" className="MSTestProject.UnitTest1" name="TestMethod1" />
    </UnitTest>
    <UnitTest name="DataRowTestMethod (1)" storage="c:\src\mstest-playground\mstestproject\mstestproject\bin\debug\net6.0\mstestproject.dll" id="fbcb88c4-070c-4e3c-a6fc-fac91aafc616">
      <Execution id="61cba81d-ea31-49ba-90e2-392492ded38a" />
      <TestMethod codeBase="C:\src\mstest-playground\MSTestProject\MSTestProject\bin\Debug\net6.0\MSTestProject.dll" adapterTypeName="executor://mstestadapter/v2" className="MSTestProject.UnitTest1" name="DataRowTestMethod" />
    </UnitTest>
    <UnitTest name="DataRowTestMethod (0)" storage="c:\src\mstest-playground\mstestproject\mstestproject\bin\debug\net6.0\mstestproject.dll" id="face297a-edd7-48ce-9aaa-65a72fe2c332">
      <Execution id="c953f320-0876-425b-9cef-1fa6a3fbfd2f" />
      <TestMethod codeBase="C:\src\mstest-playground\MSTestProject\MSTestProject\bin\Debug\net6.0\MSTestProject.dll" adapterTypeName="executor://mstestadapter/v2" className="MSTestProject.UnitTest1" name="DataRowTestMethod" />
    </UnitTest>
    <UnitTest name="DataRowTestMethod (2)" storage="c:\src\mstest-playground\mstestproject\mstestproject\bin\debug\net6.0\mstestproject.dll" id="22390aeb-07a5-4cbe-bdcd-2fe6c90b72f7">
      <Execution id="79730bea-94e1-42e5-81ee-544788341d60" />
      <TestMethod codeBase="C:\src\mstest-playground\MSTestProject\MSTestProject\bin\Debug\net6.0\MSTestProject.dll" adapterTypeName="executor://mstestadapter/v2" className="MSTestProject.UnitTest1" name="DataRowTestMethod" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="fbcb88c4-070c-4e3c-a6fc-fac91aafc616" executionId="61cba81d-ea31-49ba-90e2-392492ded38a" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="face297a-edd7-48ce-9aaa-65a72fe2c332" executionId="c953f320-0876-425b-9cef-1fa6a3fbfd2f" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="22390aeb-07a5-4cbe-bdcd-2fe6c90b72f7" executionId="79730bea-94e1-42e5-81ee-544788341d60" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestEntry testId="6349b831-af4e-4046-87c2-ec8e3ecffc93" executionId="d6e08bfa-b602-497a-8477-3d0711620467" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="4" executed="4" passed="4" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
  </ResultSummary>
</TestRun>

What’s important to note is:

• the total number of tests is 4.
• the data row entries are flatten and have different test IDs.
• the TestDefinitions contains 4 entries

Azure DevOps

Azure DevOps is parsing the trx file and using the results to populate their UI. For the previous trx file, display will be:

• 4/4 reported for the number of tests because they are reading the TestDefinitions entries of the trx.
• All datarow items are displayed flat (DataRowTestMethod (0), DataRowTestMethod (1), DataRowTestMethod (2)) at the same indent level as TestMethod1.

Reasoning for the change in 2.2.4

This change was done to overcome some issues on MSTest (e.g. if datarows are displayed as children they cannot have a different test ID and so are not expanded in Visual Studio - leading to impossibility to run/debug only 1 case) and also to have a better parity with results of NUnit and xUnit test frameworks.

What’s next

I am going to start a discussion with AzDo team to improve the UX by allowing some grouping by test while keeping the test count, test filter and other functionalities working properly.

Hey there! Sorry the messages weren’t super clear.

The issue raised in this ticket wasn’t linked to a problem in MSTest per say but it was linked to some change of behavior in MSTest (it should not have been done in a patch version) that wasn’t reflected on AzDo side. AzDo team updated the behavior on their side and we were waiting for it to be deployed (according to @dogirala it was done 2 days ago).

Regarding on-prem instances, @dogirala could you please let us know what’s the process?

@charlesdwmorrison Could you confirm that you are using non on-prem AzDo?

I will make some more manual tests to confirm issue is correctly fixed and will post my results here. In the meantime, I will reopen the ticket.

The fix is deployed now. All deployments are completed

@Evangelink , @charlesdwmorrison I see the fix has been deployed but somehow it didn’t reflect in microsoft which is strange. For microsoft org I still see old PTR task version which is 2.203.0, ideally it should be 2.210.0. I am checking with my team and concerned deployments team. Will let you know as soon as I find out something on this. Other ex pipeline: