azure-pipelines-tasks: VS Test Retry Has Stopped Working On Azure Pipeline
We’ve noticed in the last few days that a couple of our VS Test runs no longer retries on failures, and upon retry, it is unable to find the name spaces for those tests.
The tests are written in SpecFlow 3.9.74. We were using an old Test SDK (16.5) but on upgrade to 17.3.2, we still have this issue. We are running on Windows-Latest (2022). Tried with 2019 but had the same issue as well. We did not use VS Installer first, but I’ve tried with that, and we still have the issue where the retries no longer work
We run using the below YAML code:
- task: VSTest@2 displayName: 'VsTest - testAssemblies' inputs: testAssemblyVer2: | **\JourneyTests.dll !**\*TestAdapter.dll !**\obj\** rerunFailedTests: true rerunFailedThreshold: 35 rerunMaxAttempts: 2 testRunTitle: '${{ parameters.Environment }} - Journey Tests' searchFolder: $(PIPELINE.WORKSPACE)\${{ parameters.PipelineResource }}\${{ parameters.ArtifactResource }} ${{ if ne(parameters.CategoryFilter, '') }}: testFiltercriteria: ${{ parameters.CategoryFilter }} env: ${{ if eq(parameters.UseMocks, true) }}: UseMocks: true ${{ else }}: UseMocks: false
In our logs, you can see it start the retry attempt, then fail as it is unable to match the name space
No test matches the given testcase filter FullyQualifiedName=JourneyTests.Features.HappyPathFeature.New user sign up journey
.
This seems to have changed sometime between Tuesday of this week and Wednesday of this week. We run these tests on a schedule, so there was no code change between the runs on Tuesday and Wednesday.
Tue: Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220626-01 (x64)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET Core 3.1.29)
DTA Execution Host 19.210.32906.4
Wed: Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220626-01 (x64)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET Core 3.1.29)
DTA Execution Host 19.204.32418.1
Environment
- Azure Pipelines
- Windows 2022
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 7
- Comments: 60 (22 by maintainers)
Any update on this?
As for a workaround, referencing the task version explicitly in the YAML file works, i.e. instead of just
write
Any update?
@vinayakmsft - any updates?
This is caused by a VSTest task upgrade from 2.205.0 to 2.210.0, done in #16853. It broke handling of tests with a different/custom display name and fully qualified name.
It will not help to update or change the versions of Visual Studio, testing framework or build agents. The VSTest@2/3 tasks need to be fixed (they reference a newer dependency which broke backwards compatibility).
When tests are simply run without extra options, VSTest.Console.exe is executed directly, the test results are published by the agent to the pipeline and that’s it.
However, if retries are enabled, a closed source “DtaExecutionHost” component is put in between. It runs VSTest.Console.exe potentially multiple times, parses the output .trx files itself, publishes that to the pipeline, determines the failing ones back through some REST API, handles the reruns, and so on.
This DtaExecutionHost got upgraded from 19.204.32418.1 to 19.210.32906.4 during the upgrade of the VSTest ADO task. The way how a dependency “Microsoft.TeamFoundation.TestClient.PublishTestResults.dll” inside the DtaExecutionHost parses the .trx files changed, and it broke the test reruns for customers.
For example, for a parametrized test case with the following definition
The .trx file will have something like this:
Note the
<UnitTest name="TestMethod (One)" ...
has spaces and parentheses. But<TestMethod className="className="Example.Project.Tests.TestClass" name="TestMethod" ...
does not.Now the change in DtaExecutionHost:
This starts overriding the test case name from the
name
attribute on the<UnitTest/>
. Before:automatedTestName = TestMethod["className'"] + "." + TestMethod["name"]
After:automatedTestName = TestMethod["className"] + "." + (UnitTest["name"] ?? TestMethod["name"])
In practice, instead of making the rerun as:
VSTest.Console.exe /TestCaseFilter:"FullyQualifiedName=Example.Project.Tests.TestClass.TestMethod"
and unfortunately rerunning all variants of the test, but at least working, it now results in:VSTest.Console.exe /TestCaseFilter:"FullyQualifiedName=Example.Project.Tests.TestClass.TestMethod (One)"
Which does not work at all.It is wrong for two reasons:
Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
For Ben above, it is even simpler - just a test display name with spaces, resulting in
FullyQualifiedName=JourneyTests.Features.HappyPathFeature.New user sign up journey
instead of a proper method name. Again resulting in no test being found to retry.Hi @dogirala, no worries. Unfortunately, I do not know the hostname of our Azure DevOps instance, region is West Europe.
@SeMuell, I’ve tried in local it is working as expected without the customBatchSize. , will update here once vstest is deployed
I clarified this with @dogirala and the problem turned out to be in our custom
ITestDataSource::GetDisplayName()
implementation, which was formatting parameterized test names asMethodName(parameters)
(i.e. without a space), which was different from e.g. the standard[DataTestMethod]
, which usesMethodName (parameters)
(i.e. with a space before the opening parenthesis).After changing the display name to match the usual style, everything works fine and the VSTest task correctly retries only the single relevant test case (by using a combined, correctly escaped filter, e.g.
(FullyQualifiedName=Trouter.FunctionalTests.Test_ReconnectTtl_LongPoll&Name=Test_ReconnectTtl_LongPoll \(v3\))
). Thanks for the fix!Adding the observations from the above error, for dynamic tests or tests with custom displayname, please follow the same naming as standard [DataTestMethod] to resolve the above issue. Something like the below overload should work. Please reach out if issue is still observed
’
’
Thanks for the update. The new task version does not seem to have the desired effect on our pipeline.
Before (v2.224.0):
After (v2.227.0):
Should be fixed in VSTest@2.225.0 after #18760 gets merged
One month later, what is the status here?
Yeah I also didn’t know that version pinning worked to the last digit, very fortunate and useful. Thanks for the confirmation from elsewhere!