vstest: Integration of XUnit and TeamCity is broken

FYI I’ve found that using this argument instead means that test verbosity is increased to that required but build verbosity isn’t, so then the behaviour is actually like-for-like:

--logger:"console;verbosity=normal"

@martincostello I just bumped into the same issue! Saw your other tickets as well. I just upgraded from 2.1.101 to 2.1.300 and was getting no tests on TeamCity as well. I agree that this is a breaking change and I don’t like having to pass more arguments than before. It works passing --logger:"console;verbosity=normal" but… was better before IMHO.

Wow. I tried --verbosity normal instead of --verbosity Normal and it worked… dotnet test should probably not require that to be case sensitive, or at least output a warning about an invalid command line parameter.

@NikolayPianikov In the mean time, Teamcity .NET build agent needs to use lowercase variables for the verbosity option.

TeamCity has a lot of clients and we treat them with respect. Many of them have already established CI processes and we must not allow any things to make their lives more difficult. When developers creating their products they are using VS Studio or Rider or something else and I know few of them who are really running tests from the command line after that. Of course console tools should be comfortable for users in the command line but we should not forget about an automation of build/test/deploy process. Thus CI services are no less important customers.

Yes we thought about using pipes to reliable send an information about tests to CI. But this scenarion is not so transparent. In the case of stdOut customers can control and trace information exchange between all parts of CI and to solve a lot of issues independently. For pipes all this is much more difficult.

Another thing what we could do is to the remove direct TeamCity support in xunit, because of this is not working reliably now. In this case TeamCity and xunit will avoid conflicts with information about tests. From my point of view your team and xunit team could discuss it.

TeamCity service messages are useful for a set of scenarios like gathering statistics, code coverage, code inspections, attaching artifacts, build process’s flow control. And as @martincostello mentioned it provides real-time feedback to the TeamCity server and users can save a lot of time.

TeamCity sends service messages when:

• user runs his build under TeamCity using .NET CLI Plugin (TeamCity specifies additional test adapter)
• user manually adds the reference to TeamCity.VSTest.TestAdapter into test project. In this case the TeamCity integration is working from any TeamCity runners (for example command line runner).

xunit (for dotnet core version only) sends TeamCity service messages when it finds the TEAMCITY_PROJECT_NAME environment variable for its proccess. It’s like a way to determine if the xunit tests are runing under TeamCity server (to not publish TeamCity service messages from the users’ command line). For the similar cases TeamCity is watching for output here and when it finds some serviece message it turn off its own propagation of service messages to avoid a duplication for test information.

There is an intresting moment: when verbosity level is quiet and users cannot see TeamCity service messages created by xunit and sended by console logger this code still RECEIVES notifications about actually filtered service messages.

Summarizing, TeamCity cannot process the case with xunit for dotnet core properly, meanwhile other cases are working as expected.

To minimize the consequences we have made verbosity levels as ‘normal’ by default for .NET CLI Plugin

For TeamCity.VSTest.TestAdapter we have made verbosity levels as ‘normal’ by default using the paramer VSTestVerbosity, excepted the case when user manually specifies other verbosity level.

But anyway it does not look very good

With the console, TeamCity streams the test results to CI jobs in real time as they run. With .trx, don’t you have to wait for the job to complete to get all the results at once from the XML file? Knowing a long-running test job is failing early on, rather than having to wait for the whole job to complete, is much more valuable.

@NikolayPianikov Unfortunate that this came as a breaking change for you guys, but glad that you have a solution in place now. As I mentioned earlier, this change in behavior is actually a fix we made to ensure we are consistent. The current behavior is by design.

The solution you guys have now seems appropriate for specification. This change is not a take back as such, because logger verbosity can be configured for detailed logging. Please let me know if you disagree. Thanks.

@zone117x @martincostello thank you for info, done. Try updating to new version

To illustrate this change, here is a screenshot from my application’s CI build in TeamCity.

Build 1.0.1105 uses --verbosity=normal - the test results are shown. Build 1.0.1104 uses the previous behaviour of just running dotnet test without any explicit verbosity arguments - no test results are shown at all.

From the perspective of a TeamCity user, #1396 was a breaking change.

Passing --verbosity=normal as an argument to dotnet test outputs test results to TeamCity (2017.1.4) as it did in previous versions of the SDK when using 2.1.300 of the .NET Core SDK.

@smadala, @martincostello please take a look at this problem. In SDK 2.1.104 all works fine, since 2.1.200 we’re able to reproduce it.