azure-pipelines-tasks: VsTest task fails even when all tests passed

We have a project where VsTest task consistently fails even when all tests passed.

In the log, I see this:

2017-01-22T16:22:52.4050809Z Total tests: 150. Passed: 149. Failed: 0. Skipped: 1.
2017-01-22T16:22:52.4050809Z Test Run Successful.
2017-01-22T16:22:52.4050809Z Test execution time: 17.1337 Minutes
2017-01-22T16:22:52.5342314Z ##[debug]rc:0
2017-01-22T16:22:52.5342314Z ##[debug]success:false
2017-01-22T16:22:52.5342314Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2017-01-22T16:22:52.5461701Z ##[error]Error: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 0

When I download and open the .trx file from the build agent and open in Visual Studio, it says that all tests passed (one is skipped, because it is marked using the Ignore attribute which is correct).

The version of VsTest task is 2.0.5 (C:\BuildAgent\tasks\VSTest\2.0.5)

We are using the default configuration, no special options. We have installed both VS 2015 and VS 2017 RC on the build agent.

logs_7207.zip

TFSBA_TFSBA4WS12R2 2017-01-22 16_06_05.zip

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 10
Comments: 18 (6 by maintainers)

Commits related to this issue

Update vstest.ts `vstest.console.exe` writes to stderr when there are inconclusive or ignored tests which causes this task to fail, even if all tests are successful and `vstest.console.exe` returns c... — committed to ChristopherHaws/vso-agent-tasks by ChristopherHaws 7 years ago
Ignore stderr errors when running vstest.console.exe `vstest.console.exe` writes to stderr when there are inconclusive or ignored tests which causes this task to fail, even if all tests are successfu... — committed to ChristopherHaws/vso-agent-tasks by ChristopherHaws 7 years ago

Most upvoted comments

@nigurr I am using the vstest task v2 that uses the Visual Studio Test Platform Installer of version 15.7.2. All tests pass and there are no skipped or inconclusive tests but the task is still marked as red. The tests are using NUnit3. Here is the log (replaced some sensitive information like machine names, fileshares and product info):

2018-06-01T07:25:14.2089314Z ##[section]Starting: VsTest - testAssemblies
2018-06-01T07:25:14.2095887Z ==============================================================================
2018-06-01T07:25:14.2096159Z Task         : Visual Studio Test
2018-06-01T07:25:14.2096684Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).
2018-06-01T07:25:14.2097110Z Version      : 2.5.12
2018-06-01T07:25:14.2097330Z Author       : Microsoft Corporation
2018-06-01T07:25:14.2097615Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-06-01T07:25:14.2097909Z ==============================================================================
2018-06-01T07:25:15.1285017Z Run the tests locally using vstest.console.exe
2018-06-01T07:25:15.1288060Z ========================================================
2018-06-01T07:25:15.1289080Z Test selector : Test assemblies
2018-06-01T07:25:15.1289617Z Test assemblies : **\*.IntegrationTests.dll
2018-06-01T07:25:15.1290110Z Test filter criteria : null
2018-06-01T07:25:15.1290716Z Search folder : G:\ProductName Integration Test Share\Tests
2018-06-01T07:25:15.1291334Z Run settings file : \\filesharefqdn\Drop$\ProductName\Releases\8554\8890\Tasks\IT.runsettings
2018-06-01T07:25:15.1292025Z Run in parallel : false
2018-06-01T07:25:15.1292485Z Run in isolation : false
2018-06-01T07:25:15.1296667Z Path to custom adapters : null
2018-06-01T07:25:15.1297154Z Other console options : null
2018-06-01T07:25:15.1298281Z Code coverage enabled : false
2018-06-01T07:25:15.1299296Z Rerun failed tests: false
2018-06-01T07:25:16.1996907Z VisualStudio version selected for test execution : toolsInstaller
2018-06-01T07:25:16.3221608Z ========================================================
2018-06-01T07:25:16.8557669Z [command]D:\_Builds\buildagentmachine.01\_tool\VsTest\15.7.2\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe @D:\_Builds\buildagentmachine.01\_temp\ee439541-656c-11e8-82c3-95625c763305.txt
2018-06-01T07:25:16.9204325Z Microsoft (R) Test Execution Command Line Tool Version 15.7.2
2018-06-01T07:25:16.9204856Z Copyright (c) Microsoft Corporation.  All rights reserved.
2018-06-01T07:25:16.9260292Z vstest.console.exe 
2018-06-01T07:25:16.9261414Z "path/to/testDLL.dlll"
2018-06-01T07:25:16.9272975Z /Settings:"\\serverFQDN\Drop$\ProductName\Releases\8554\8890\Tasks\IT.runsettings"
2018-06-01T07:25:16.9273811Z /logger:"trx"
2018-06-01T07:25:16.9274613Z /TestAdapterPath:"G:\Product Name Integration Test Share\Tests"
2018-06-01T07:25:17.1423229Z Starting test execution, please wait...
2018-06-01T07:25:20.9896664Z Multiple versions of same extension found. Selecting the highest version.
2018-06-01T07:25:20.9897288Z   NUnit3.TestAdapter : 3.10.0.21
2018-06-01T07:25:23.4152009Z NUnit Adapter 3.10.0.21: Test execution started
Running all tests in path/to/testDLL.dll
2018-06-01T07:35:29.0663367Z NUnit3TestExecutor converted 45 of 45 NUnit test cases
.... (test results)
NUnit Adapter 3.10.0.21: Test execution complete
2018-06-01T07:38:07.3388517Z NUnit VS Adapter 2.0.0.0 executing tests is started
2018-06-01T07:38:08.0838379Z Exception NUnit.Core.UnsupportedFrameworkException, Exception thrown executing tests in path/to/testDLL.dll
2018-06-01T07:38:09.6946217Z NUnit VS Adapter 2.0.0.0 executing tests is finished
2018-06-01T07:38:10.0989892Z M
Total tests: 890. Passed: 890. Failed: 0. Skipped: 0.
2018-06-01T07:38:16.0669743Z Test Run Failed.
2018-06-01T07:38:16.0682091Z Test execution time: 12.9580 Minutes
2018-06-01T07:38:16.0926380Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2018-06-01T07:38:16.1082348Z ##[error]Error: D:\_Builds\buildmachine.01\_tool\VsTest\15.7.2\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 1
2018-06-01T07:38:16.9141667Z ##[error]VsTest task failed.

P.S. I noticed that this seems to be due to our test runsettings still pointing to an old NUnitv2 test adapter path but the point is that whatever happens, only the test results should be used to mark it red or not. If something unexpected like this happens, you can still mark it as orange to warn about it.

bergmeister on Jun 1, 2018

I have the same problem.

all tests passed, 2 skipped tests, but agent determines that the build fails. Using NUnit 3.6.0 and NUnitTestAdapters 3.7.0

It does because Skipped tests generate error messages when running 2>error.text

DOMZE on Feb 1, 2017

Finally, I found the issue. Well mine is a bit unexpected. Some very capable developer decided to check in the bin release folder into source control. The checked in folder contained nunit 2 test adapter. Even though it’s in a totally different folder as my nunit 3 tests, somehow it vstest decides to try and use the version 2 adapter to run my nunit 3 tests. Even though within the same folder, there is nunit 3 test adapter dlls also. I can finally stop pulling my hair out now. 😦

So my final solution was to delete those old bin folders and get on with my life.

kfry88 on Jun 3, 2019

@nigurr Your link does not explain how we can use the vstest task with skipped tests and have the task only fail on failed tests. The PR above by @ChristopherHaws (#3558) could have helped you to achieve this I have a case where there are no skipped tests but it still marks it as red because vstestconsole.exe returns an exit code of 1.

bergmeister on May 31, 2018