vstest: dotnet test is incompatible with MSBuild node reuse
Description
In the .NET SDK 2.1.300, MSBuild on .NET Core will introduce support for node reuse. This feature enable faster cycle times by keeping MSBuild worker processes alive after the first build ends, allowing them to be reused “hot” by the second and subsequent builds.
Unfortunately, this is incompatible with the output strategy used by VSTestTask, which depends on child processes of MSBuild being attached to the stdout stream of the entry-point dotnet test process. That can’t be the case when vstest.console may be run from an MSBuild that has no stdout stream.
On a CLI with the new MSBuild, output from VSTest does not appear on the console.
Steps to reproduce
This is the root cause of the test failures in https://ci.dot.net/job/dotnet_cli/job/release_2.1.2xx/job/release_windows_nt_x64_prtest/41/
On a CLI with the new MSBuild, run dotnet test.
Expected behavior
Test output is displayed from dotnet test regardless of MSBuild internals. From preview1:
s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti>"C:\Program Files\dotnet\dotnet.exe" test
Build started, please wait...
Build completed.
Test run for s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti\bin\Debug\net46\VSTestDesktopAndNetCore.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 15.6.0-preview-20180109-01
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Failed VSTestFailTest
Error Message:
Assert.Fail failed.
Stack Trace:
at TestNamespace.VSTestTests.VSTestFailTest()
Total tests: 3. Passed: 2. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 1.8794 Seconds
Test run for s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti\bin\Debug\netcoreapp2.1\VSTestDesktopAndNetCore.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.6.0-preview-20180109-01
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
Failed VSTestFailTest
Error Message:
Assert.Fail failed.
Stack Trace:
at TestNamespace.VSTestTests.VSTestFailTest() in s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti\Tests.cs:line 19
Failed VSTestFailTestNetCoreApp
Error Message:
Assert.Fail failed.
Stack Trace:
at TestNamespace.VSTestTests.VSTestFailTestNetCoreApp() in s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti\Tests.cs:line 31
Total tests: 3. Passed: 1. Failed: 2. Skipped: 0.
Test Run Failed.
Test execution time: 1.7366 Seconds
s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti>
Actual behavior
Test output from dotnet test is lost; the build fails without error.
s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti>dotnet test
Build started, please wait...
Build completed.
s:\cli\bin\3\win-x64\test\dotnet-test.Tests\1\VSTestMulti>
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 5
- Comments: 19 (13 by maintainers)
Commits related to this issue
- Keep stdout for test execution Work around https://github.com/Microsoft/vstest/issues/1503 by using the MSBuild escape hatch variable MSBUILDNODEWINDOW and ensuring that tests don't run in a disconne... — committed to rainersigwald/cli by rainersigwald 6 years ago
- Keep stdout for test execution Work around https://github.com/Microsoft/vstest/issues/1503 by using the MSBuild escape hatch variable MSBUILDNODEWINDOW and ensuring that tests don't run in a disconne... — committed to rainersigwald/cli by rainersigwald 6 years ago
- Keep stdout for test execution Work around https://github.com/Microsoft/vstest/issues/1503 by using the MSBuild escape hatch variable MSBUILDNODEWINDOW and ensuring that tests don't run in a disconne... — committed to rainersigwald/cli by rainersigwald 6 years ago
- Escape hatch: share stdout with node children The VSTest runner emits log messages to its stdout. So far, in .NET Core, that has resulted in them getting piped all the way up through the worker MSBui... — committed to rainersigwald/msbuild by rainersigwald 6 years ago
- Keep stdout for test execution Work around https://github.com/Microsoft/vstest/issues/1503 by using the MSBuild escape hatch variable MSBUILDENSURESTDOUTFORTASKPROCESSES and ensuring that tests don't... — committed to rainersigwald/cli by rainersigwald 6 years ago
- Escape hatch: share stdout with node children (#3130) The VSTest runner emits log messages to its stdout. So far, in .NET Core, that has resulted in them getting piped all the way up through the wo... — committed to dotnet/msbuild by rainersigwald 6 years ago
- Keep stdout for test execution Work around https://github.com/Microsoft/vstest/issues/1503 by using the MSBuild escape hatch variable MSBUILDENSURESTDOUTFORTASKPROCESSES and ensuring that tests don't... — committed to rainersigwald/cli by rainersigwald 6 years ago
- Invoke dotnet vstest directly instead of using the VSTest target Attempt to workaround Microsoft/vstest#1503 which causes test failure output to be swallowed — committed to aspnet/Universe by deleted user 6 years ago
- Fix test output during CI. When tests run in CI, we are not displaying the test output to the console. So if a test fails, and for some reason the .trx file isn't parsed correctly, it is impossible t... — committed to eerhardt/machinelearning by eerhardt 6 years ago
- Fix test output during CI. (#652) When tests run in CI, we are not displaying the test output to the console. So if a test fails, and for some reason the .trx file isn't parsed correctly, it is impos... — committed to dotnet/machinelearning by eerhardt 6 years ago
- Run tests using arcade - Disable node reuse due to https://github.com/Microsoft/vstest/issues/1503 - Ensure that stdout works correctly: https://github.com/Microsoft/vstest/issues/1503#issuecomme... — committed to sbomer/linker by sbomer 5 years ago
What’s holding it up do you think?
I am already using
dotnet msbuild. I am assuming you mean after #2702 is merged?Yes, it is. As a workaround you can set the environment variable
MSBUILDENSURESTDOUTFORTASKPROCESSES=1in addition to passing/nodereuse:false. That changes the way MSBuild creates child processes to force them to share a stdout stream with the parent.The
dotnetcli uses this workaround when you invokedotnet test. @FlukeFan the build is still parallel in thedotnet testcase, but it doesn’t reuse MSBuild worker nodes, so there’s slightly more overhead than usual.