azure-pipelines-tasks: VsTest 2.3.17 - Test Impact Analysis Failing

Environment

TFS On-Premise 2018 RTM (16.122.27102.1)

Private Agent (v2.122.1) on Windows Server 2012 R2

VsTest 2.3.17

Issue Description

We have a vNext build that builds a C# project and then runs unit tests using the VsTest task. We are wanting to use Test Impact Analysis (TIA) but are having a couple of problems.

Firstly - The build that normally takes 15 minutes takes 115 minutes. I can expect it to take longer, but is this realistic for ~800 unit tests?

Secondly and more importantly, after running the tests and gathering the testimpact.xml files it seems to crash when using TestSelector.exe.

Error logs

6_Test Assemblies.txt 6_TestSelector.log 6_eaeda910-f606-11e7-af9a-63fe0f449198.txt

I’ve attached a snippet of the log below:

2018-01-10T14:49:28.0261411Z ##[debug]C:\TfsAgents\AgentA\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.3.17\TestSelector\TestSelector.exe arg: UpdateTestResults
2018-01-10T14:49:28.0261411Z ##[debug]Build.BuildId=38430
2018-01-10T14:49:28.0261411Z ##[debug]System.TeamFoundationCollectionUri=http://tfs/DefaultCollection/
2018-01-10T14:49:28.0261411Z ##[debug]System.TeamProject=STANMIS
2018-01-10T14:49:28.0261411Z ##[debug]SystemVssConnection auth param AccessToken = ********
2018-01-10T14:49:28.0261411Z ##[debug]AGENT.VERSION=2.122.1
2018-01-10T14:49:28.0261411Z ##[debug]VSTS_HTTP_RETRY=undefined
2018-01-10T14:49:28.0261411Z ##[debug]VSTS_HTTP_TIMEOUT=undefined
2018-01-10T14:49:28.0261411Z ##[debug]System.Debug=true
2018-01-10T14:49:28.0261411Z ##[debug]exec tool: C:\TfsAgents\AgentA\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.3.17\TestSelector\TestSelector.exe
2018-01-10T14:49:28.0261411Z ##[debug]arguments:
2018-01-10T14:49:28.0261411Z ##[debug]   UpdateTestResults
2018-01-10T14:49:28.0261411Z [command]C:\TfsAgents\AgentA\_work\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.3.17\TestSelector\TestSelector.exe UpdateTestResults
2018-01-10T14:49:29.7448898Z 
2018-01-10T14:49:29.7448898Z 
2018-01-10T14:49:29.7448898Z Exception : TestManagement.TestResults.UsePublishTestResultsLibraryInVSTestTask
2018-01-10T14:49:29.7448898Z ***************************** Section 'Update test results' Starting *****************************
2018-01-10T14:49:30.8542669Z ##[debug]Processed: ##vso[telemetry.publish area=TestExecution;feature=TestSelector]{"Command":"UpdateTestResults","definitionrunid":38430,"context":"CI","GetTestCaseResults:PerfCounter":551.19490000000008,"ExceptionType":"System.AggregateException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089","ExceptionMessage":"One or more errors occurred.","ExceptionStack":"   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\r\n   at System.Threading.Tasks.Task`1.get_Result()\r\n   at Microsoft.TestFoundation.TestImpact.TestSelector.CommandUpdateTestResults.GetFeatureFlagValue() in E:\\v2.0\\A8\\_work\\21\\s\\Ta\\Tasks\\TestImpact\\TestSelector\\Commands\\Command\\CommandUpdateTestResults.cs:line 206\r\n   at Microsoft.TestFoundation.TestImpact.TestSelector.CommandUpdateTestResults.Run() in E:\\v2.0\\A8\\_work\\21\\s\\Ta\\Tasks\\TestImpact\\TestSelector\\Commands\\Command\\CommandUpdateTestResults.cs:line 139\r\n   at Microsoft.TestFoundation.TestImpact.TestSelector.Program.RunCommand(String commandName, String[] args) in E:\\v2.0\\A8\\_work\\21\\s\\Ta\\Tasks\\TestImpact\\TestSelector\\Program.cs:line 44","InnerExceptionType":"Microsoft.VisualStudio.Services.Common.VssServiceException, Microsoft.VisualStudio.Services.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","InnerExceptionMessage":"TestManagement.TestResults.UsePublishTestResultsLibraryInVSTestTask","InnerExceptionStack":"   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<HandleResponseAsync>d__51.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 893\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__49.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 836\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__46`1.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 727\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__27`1.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 0","VsTest_TaskInstanceIdentifier":"eae1c233-f606-11e7-af9a-63fe0f449198"}
2018-01-10T14:49:30.8542669Z ***************************** Section 'Update test results' Ended ********************************
2018-01-10T14:49:30.8542669Z 
2018-01-10T14:49:30.8542669Z ##[debug]Processed: ##vso[task.logissue type=warning;SubTaskName=UploadTestResults;SubTaskDuration=3359.1329070003703]
2018-01-10T14:49:30.8542669Z ##[debug]Total time taken to upload test results: 3359.1329070003703 milliseconds.
2018-01-10T14:49:30.8542669Z ##[error]
Exception : TestManagement.TestResults.UsePublishTestResultsLibraryInVSTestTask

2018-01-10T14:49:30.8542669Z ##[debug]Processed: ##vso[task.issue type=error;]%0AException : TestManagement.TestResults.UsePublishTestResultsLibraryInVSTestTask%0D%0A

Exception specific information:

ExceptionType: 
System.AggregateException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

ExceptionMessage:
One or more errors occurred.

ExceptionStack:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n  
 at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\r\n  
 at System.Threading.Tasks.Task`1.get_Result()\r\n   
at Microsoft.TestFoundation.TestImpact.TestSelector.CommandUpdateTestResults.GetFeatureFlagValue() in E:\\v2.0\\A8\\_work\\21\\s\\Ta\\Tasks\\TestImpact\\TestSelector\\Commands\\Command\\CommandUpdateTestResults.cs:line 206\r\n   
at Microsoft.TestFoundation.TestImpact.TestSelector.CommandUpdateTestResults.Run() in E:\\v2.0\\A8\\_work\\21\\s\\Ta\\Tasks\\TestImpact\\TestSelector\\Commands\\Command\\CommandUpdateTestResults.cs:line 139\r\n   
at Microsoft.TestFoundation.TestImpact.TestSelector.Program.RunCommand(String commandName, String[] args) in E:\\v2.0\\A8\\_work\\21\\s\\Ta\\Tasks\\TestImpact\\TestSelector\\Program.cs:line 44",

"InnerExceptionType":"Microsoft.VisualStudio.Services.Common.VssServiceException, Microsoft.VisualStudio.Services.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a",

"InnerExceptionMessage":"TestManagement.TestResults.UsePublishTestResultsLibraryInVSTestTask",

"InnerExceptionStack":"   
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<HandleResponseAsync>d__51.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 893\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__49.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 836\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__46`1.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 727\r\n
--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__27`1.MoveNext() in E:\\v2.0\\A8\\_work\\21\\s\\Vssf\\Client\\WebApi\\VssHttpClientBase.cs:line 0

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 17 (9 by maintainers)

Most upvoted comments

We chatted on the issue and looks like the profiler we have is adding overhead for one of the build definition. I have suggested a way to @psyvision which he is going to try out. The time came down to 2.1 minutes!. below is the chnage i asked

You need to do follow changes

Enable Trigger build as scheduled build for your definition and keep it on week end or your non office hours
Change vstest task input “Number of builds after which all tests should be run to $(FullIterationInterval)

Add FullIterationInterval as one of the build variables and make the value as 50 or value which you want to have. Add one more variable call it tia.disabletiadatacollector and set it to true 3. Add a powershell task just above visual studio test task and have this content in that Write-Host “##vso[task.setvariable variable= FullIterationInterval;]1” Write-Host “##vso[task.setvariable variable= tia.disabletiadatacollector;]false” 4. In above powershell task choose custom condition

So what will happen is by default TIA profiler wont be on. It will be on only because of the powershell script. And the powershell script is supposed to be run only for scheduled build.

acesiddhu on Feb 8, 2018

If you upload a task manually, the system returns the in the box versions unioned with the custom one you uploaded. If you use tfx to then delete that task, we delete the overlay. You cannot delete the in the box versions we upload on upgrade so that works fine.

Regarding support, it’s not that it’s not supported. It’s more of you’re kind of on your own to build correctly and upload. We don’t release an official “update pack” even though we’ve considered it.

Also, always make sure it’s a release branch since master is unshipped code.

hth

bryanmacfarlane on Feb 6, 2018