sdk: Regression: `dotnet test` hangs for 15 minutes after test run completed on mac and linux

I can confirm that this still happens on .NET Core 3.1

ubuntu@VM-0-4-ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

root@195d912bb800:/# dotnet --version
3.1.100

Test run for ***********************(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.                      <---- hangs forever

-nodereuse:false doesn’t work either.

Hello Guys, I tried all options said here, but nothing work.

I solves my problem (happening only in azure devops integration) turning off parallel test executing

In my test project > Add folder Properties > Add AssemblyInfo.cs

using Xunit;

[assembly: CollectionBehavior(DisableTestParallelization = true)]

I can confirm that this still happens on .NET Core 3.1

ubuntu@VM-0-4-ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

root@195d912bb800:/# dotnet --version
3.1.100

Test run for ***********************(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.                      <---- hangs forever

-nodereuse:false doesn’t work either.

This the exact spot where my test run also hangs.

One interesting point I noticed is that the same code without changes runs in Visual Studio for Mac (the very first test method runs ok, and the others take 2+ minutes each - pure unit tests with mocking - so not even hitting the db). Not sure if that can help solve the issue. dotnet version: 3.1.201

I am using Xunit and was able to fix this issue by adding the following to csproj:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />

FWIW

I’m having a similar situation at github actions with setup .NET action, using XUnit + .NET6

I am still seeing this behavior on Azure DevOps with .NET Core 3.1.301 on a Windows Hosted Agent with VS2019

running dotnet test within a powershell script causes it to hang at the end of the execution. willing to share build and repro details.

Eg: Get’s stuck here for 40+ minutes

CC: @rainersigwald @peterhuene @livarcocc

I just want to chime in and say that I’m experiencing this issue as well. I haven’t been able to reproduce it reliably yet, but I’ll keep trying and once I do, I’ll post a link to a repo here.

The scenario I have is a unit test that is testing some MSBuild targets files. This involves calling dotnet build during the unit test. This is basically what it does:

1. Write a project file that uses the targets to disk.
2. Compile the project using dotnet build.
3. Look at the output from the build and assert some stuff.

When running this via dotnet test, it sometimes hangs for 15 minutes after the tests complete (the test duration reported is only a few seconds), but sometimes it doesn’t hang at all.

@Evengard Try the --blame and --blame-hang flags to help pinpoint the hanging tests. It creates a memory dump on hang timeout that helped me find out that it was one of my hosted services that was still alive.

https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test

Filter flag is optional, but useful to limit the tests you want to run.

dotnet test "$test_project" \
  --no-build \
  --filter "$filter" \
  --logger trx \
  --logger "console;verbosity=detailed" \
  --verbosity "normal" \
  --results-directory TestResults \
  --blame \
  --blame-hang \
  --blame-hang-timeout 1min \
  --diag TestResults/vstest_diagnostics.log

Seems like passing -v:quiet to dotnet test (not needed for dotnet build, only for dotnet test - at least in my experience) alleviates the issue somehow… No ideas how, but well, it works. I guess the solution above was the same - just setting the verbosity to quiet.

Also from here seems like running the binary directly inside a container (eg as a dockerfile entrypoint, or in my case directly with kubectl exec inside a running kubernetes pod), not wrapping it into a bash call, seems to invoke the issue somehow, whereas wrapping it into a bash call seems to alleviate it. Maybe bash somehow helps with theese stdin/stdout streams deadlocks?

Can you please explain @ana-cozma what exactly did you change? The steps you included seem pretty standard to me, unless you have something special in the runsettings file.

Hi, I think lot of these issues are due to a bug in the Process type, I opened an issue about it here: https://github.com/dotnet/runtime/issues/51277

In summary:

The issue is that WaitForExit() will synchronously wait for the stderr/stdout to complete, the summoned process has exited, but it has childs process still running (the nodereuse processes in msbuild case).
In this case, it looks like that the stderr/stdout pipes wont close.

It looks like most of these issues happens when the dotnet sdk is summoned from another dotnet process, so with the Process class.

@reduckted You should be able to resolve that by calling dotnet build -nodereuse:false inside your test. What’s happening for you is that the outer dotnet test sets the environment variable that MSBuild uses to decide whether to attach the console or not. That is propagated down through child processes to the test process and then to the test’s child dotnet build process, which then starts with node reuse but without the safe disconnect-console node startup. Explicitly specifying “no node reuse” should cause the worker nodes to exit immediately after the build, allowing the wait-on-child-process code to complete quickly.