azure-pipelines-tasks: VS Test Retry Has Stopped Working On Azure Pipeline

Any update on this?

As for a workaround, referencing the task version explicitly in the YAML file works, i.e. instead of just

    - task: VSTest@2
# or
    - task: VSTest@3

write

    - task: VSTest@2.205.0  # Using a full version temporarily to fix https://github.com/microsoft/azure-pipelines-tasks/issues/16990
# or
    - task: VSTest@3.205.0  # Using a full version temporarily to fix https://github.com/microsoft/azure-pipelines-tasks/issues/16990

Any update?

@vinayakmsft - any updates?

This is caused by a VSTest task upgrade from 2.205.0 to 2.210.0, done in #16853. It broke handling of tests with a different/custom display name and fully qualified name.

It will not help to update or change the versions of Visual Studio, testing framework or build agents. The VSTest@2/3 tasks need to be fixed (they reference a newer dependency which broke backwards compatibility).

---

When tests are simply run without extra options, VSTest.Console.exe is executed directly, the test results are published by the agent to the pipeline and that’s it.

However, if retries are enabled, a closed source “DtaExecutionHost” component is put in between. It runs VSTest.Console.exe potentially multiple times, parses the output .trx files itself, publishes that to the pipeline, determines the failing ones back through some REST API, handles the reruns, and so on.

This DtaExecutionHost got upgraded from 19.204.32418.1 to 19.210.32906.4 during the upgrade of the VSTest ADO task. The way how a dependency “Microsoft.TeamFoundation.TestClient.PublishTestResults.dll” inside the DtaExecutionHost parses the .trx files changed, and it broke the test reruns for customers.

---

For example, for a parametrized test case with the following definition

    [TestMethod]
    [DataTestMethod]
    [DataRow("One")]
    [DataRow("Two")]
    public void TestMethod(string Parameter)

The .trx file will have something like this:

  <TestDefinitions>
    <UnitTest name="TestMethod (One)"
               storage="c:\whatever\bin\debug\net6.0\testproject.dll"
               id="df932f1e-0bdf-4fd8-b4e0-1d20f4f13e98">
      <Execution id="101c7677-781f-43dd-9240-442df2afb88d" />
      <TestMethod codeBase="C:\Whatever\bin\Debug\net6.0\TestProject.dll"
               adapterTypeName="executor://mstestadapter/v2"
               className="Example.Project.Tests.TestClass"
               name="TestMethod" />
    </UnitTest>

Note the <UnitTest name="TestMethod (One)" ... has spaces and parentheses. But <TestMethod className="className="Example.Project.Tests.TestClass" name="TestMethod" ... does not.

Now the change in DtaExecutionHost:

--- before\Microsoft.TeamFoundation.TestClient.PublishTestResults-19.204.32418.1\Microsoft.TeamFoundation.TestClient.PublishTestResults\TrxResultParser.cs	Mon Oct 03 16:56:16 2022
+++ after\Microsoft.TeamFoundation.TestClient.PublishTestResults-19.210.32906.4\Microsoft.TeamFoundation.TestClient.PublishTestResults\TrxResultParser.cs	Mon Oct 03 16:55:38 2022
@@ -229,9 +229,9 @@
 			}
 			XmlNode testResultNode = definitionNode.SelectSingleNode("./TestMethod");
 			string automatedTestName = null;
-			if (testResultNode?.Attributes["className"] != null && testResultNode.Attributes["name"] != null)
+			if (testResultNode?.Attributes["className"] != null && (definitionNode.Attributes["name"]?.Value != null || testResultNode.Attributes["name"].Value != null))
 			{
-				string testCaseName = testResultNode.Attributes["name"].Value;
+				string testCaseName = ((definitionNode.Attributes["name"]?.Value == null) ? testResultNode.Attributes["name"].Value : definitionNode.Attributes["name"].Value);
 				string className = testResultNode.Attributes["className"].Value;
 				automatedTestName = ((testCaseName.StartsWith(className) && testCaseName.Contains(".")) ? testCaseName : string.Join(".", className, testCaseName));
 				if (_chutzpahTestFileExtensions.Contains(storageExtension?.Trim(), StringComparer.OrdinalIgnoreCase) && !string.IsNullOrEmpty(storageFileName))

This starts overriding the test case name from the name attribute on the <UnitTest/>. Before: automatedTestName = TestMethod["className'"] + "." + TestMethod["name"] After: automatedTestName = TestMethod["className"] + "." + (UnitTest["name"] ?? TestMethod["name"])

In practice, instead of making the rerun as: VSTest.Console.exe /TestCaseFilter:"FullyQualifiedName=Example.Project.Tests.TestClass.TestMethod" and unfortunately rerunning all variants of the test, but at least working, it now results in: VSTest.Console.exe /TestCaseFilter:"FullyQualifiedName=Example.Project.Tests.TestClass.TestMethod (One)" Which does not work at all.

It is wrong for two reasons:

1. The paretheses are not escaped correctly Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
2. It is not a valid FullyQualifiedName, that needs really just the method FQN. Comparing the display name or whatever would need a completely different filter being made.

For Ben above, it is even simpler - just a test display name with spaces, resulting in FullyQualifiedName=JourneyTests.Features.HappyPathFeature.New user sign up journey instead of a proper method name. Again resulting in no test being found to retry.

Hi @dogirala, no worries. Unfortunately, I do not know the hostname of our Azure DevOps instance, region is West Europe.

@SeMuell, I’ve tried in local it is working as expected without the customBatchSize. , will update here once vstest is deployed

I clarified this with @dogirala and the problem turned out to be in our custom ITestDataSource::GetDisplayName() implementation, which was formatting parameterized test names as MethodName(parameters) (i.e. without a space), which was different from e.g. the standard [DataTestMethod], which uses MethodName (parameters) (i.e. with a space before the opening parenthesis).

After changing the display name to match the usual style, everything works fine and the VSTest task correctly retries only the single relevant test case (by using a combined, correctly escaped filter, e.g. (FullyQualifiedName=Trouter.FunctionalTests.Test_ReconnectTtl_LongPoll&Name=Test_ReconnectTtl_LongPoll \(v3\))). Thanks for the fix!

Adding the observations from the above error, for dynamic tests or tests with custom displayname, please follow the same naming as standard [DataTestMethod] to resolve the above issue. Something like the below overload should work. Please reach out if issue is still observed

’

   public string GetDisplayName(MethodInfo methodInfo, object[] data)
    {
        if (data != null)
        {
            var testParamString = new StringBuilder();
            testParamString.Append(methodInfo.Name);

            testParamString.Append(' (');
            var methodParameters = methodInfo.GetParameters();
            for (int i = 0; i < methodParameters.Length; i++)
            {
                testParamString.Append(data[i]);
                testParamString.Append(',');
            }
            testParamString.Length--;
            testParamString.Append(')');

            return testParamString.ToString();
        }

        return methodInfo.Name;
    }

’

Thanks for the update. The new task version does not seem to have the desired effect on our pipeline.

Before (v2.224.0):

2023-08-16T08:49:17.0628419Z Task : Visual Studio Test
2023-08-16T08:49:17.0628738Z Version : 2.224.0 
...
2023-08-16T08:49:17.9948378Z DtaExecutionHost version 19.210.32906.4. 
...
2023-08-16T09:06:29.5685456Z Failed Test_DisconnectOnLargeMessage(v4) [39 s]
...
2023-08-16T09:10:37.8506695Z **************** Rerunning failed tests for Test run 326350859 ********************* 
2023-08-16T09:10:37.8948952Z vstest.console.exe /TestCaseFilter:
        "FullyQualifiedName=Trouter.FunctionalTests.TrouterNativeNetClientFunctionalTest.Test_DisconnectOnLargeMessage(v4)"
...
2023-08-16T09:10:39.6810636Z ##[error]Incorrect format for TestCaseFilter Missing Operator '|' or '&'.
                             Specify the correct format and try again. Note that the incorrect format
                             can lead to no test getting executed. 

After (v2.227.0):

2023-08-24T19:39:22.9042215Z Task : Visual Studio Test
2023-08-24T19:39:22.9042542Z Version : 2.227.0 
...
2023-08-24T19:39:28.8259497Z DtaExecutionHost version 19.226.34002.2. 
...
2023-08-24T19:54:38.0774500Z Failed Test_MessageStore_TrySyncMode(v3) [13 s] 
2023-08-24T19:54:45.0069644Z Failed Test_MessageStore_TrySyncMode(v4) [6 s] 
...
2023-08-24T20:00:32.8905690Z **************** Rerunning failed tests for Test run 327331990 ********************* 
2023-08-24T20:00:32.9432344Z vstest.console.exe /TestCaseFilter:
        "(FullyQualifiedName=Trouter.FunctionalTests.TrouterNativeNetClientFunctionalTest.Test_MessageStore_TrySyncMode(v4))
        |(FullyQualifiedName=Trouter.FunctionalTests.TrouterNativeNetClientFunctionalTest.Test_MessageStore_TrySyncMode(v3))"
...
2023-08-24T20:00:34.9557818Z ##[error]Incorrect format for TestCaseFilter Missing Operator '|' or '&'.
                             Specify the correct format and try again. Note that the incorrect format
                             can lead to no test getting executed. 

Should be fixed in VSTest@2.225.0 after #18760 gets merged

We have taken this as our work item , maybe by next week we will update

One month later, what is the status here?

Yeah I also didn’t know that version pinning worked to the last digit, very fortunate and useful. Thanks for the confirmation from elsewhere!