nunit-console: NUnit3 Fails on DLL with no Tests, Unlike NUnit2
@hypersw commented on Fri Jun 17 2016
Some projects of our codebase are for test purposes only, though not all of them have test cases (some got only mocking infrastructure code).
With NUnit1-NUnit2, we used to submit all of the DLLs from the test area, and NUnit would run all the tests it could find.
With NUnit3, it fails with an error like this:
Errors and Failures
1) Invalid : xxx.dll
No suitable tests found in 'xxx.dll'.
Either assembly contains no tests or proper test driver has not been found.
And returns the ERRORLEVEL of FFFFFFFE.
This is a breaking change from NUnit2, from how I see it. We didn’t think up a workaround yet. Is it possible to get the previous version behavior, with an option or something?
@hypersw commented on Fri Jun 17 2016
Errorlevel looks like NUnit.ConsoleRunner.ConsoleRunner::INVALID_ASSEMBLY.
@hypersw commented on Fri Jun 17 2016
The error is probably triggered in NUnit.Engine.Services.DriverService::GetDriver.
@hypersw commented on Fri Jun 17 2016
Looks like it actually checks by referenced assemblies only rather than presence of the actual test cases. That’s something I could also check in the build script before spawning tests for each DLL. Might be useful to be able to batch-submit them though, as you’d not always have a build script capable of reading thru assembly references like in our case.
@hypersw commented on Fri Jun 17 2016
I’ve prototyped a fix at https://github.com/JetBrains/nunit/commit/ad29539dca76c87dd11e2491cf8537f01a9e38e3 for us to try nunit3 full-scale without missing test errors. Might not be a good idea doing this by default (even though this seems to be a breaking change from nunit2), but could prove useful if under an option.
@CharliePoole commented on Fri Jun 17 2016
Well, it’s 3.x versus 2.x, so it has a ton of breaking changes. Minimum “fix” for this issue would be to make sure the change is listed on the Breaking Changes page. 😃
That said, I think this is a correct change. In most usage, passing a non-test assembly to NUnit is an error of some kind. It could be a simple, ignorable error as in your case. OTOH, it could be that you are using some third-party framework without installing the driver extension for that framework. There is no way for NUnit to know what is the cause.
This has already given us a lot of trouble with the VS adapter because VS passes us all the assemblies in a solution and it’s also a problem for users who try to run the solution from the command line. We deal with the first by special checks in the adapter. The second is the subject of a separate issue (#668), which we will probably take care of by passing in some info about the source.
So, for various reasons, a package setting of “IgnoreNonTestAssemblies” is probably in the future. Once that’s available, we could add a console option to activate it.
Note: This should be worked on in conjunction with #668.
@DanHarman commented on Tue Sep 27 2016
Is there any further thinking on this? My particular scenario is passing in a glob to my teamcity nunit build step for:
**\bin\Release\*Tests.dll
As this means we don’t need to keep adding test projects into the build server manually (I’ve already failed to do this once…) However we also have mstest projects in the build which also have *Tests.dll extension, and nunit console test runner errors out as soon as it encounters one. Ideally I’d like a command line param to not error out if it is passed a dll without any tests it understands.
@CharliePoole commented on Tue Sep 27 2016
Well, as a result of earlier discussion, we accepted this as a feature request and added it to our backlog.
The next thing you can watch for is for somebody to take it on as an assignment. That could be a team member or an outside contributor who wants it badly. 😄
@rprouse commented on Tue Sep 27 2016
If we decide to change this, we should do the same in the .NET Core runner. There is a duplicate issue there with a couple of requests, nunit/dotnet-test-nunit#77
@rprouse commented on Tue Sep 27 2016
This is also a Console issue, I am going to move this issue there.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 40 (22 by maintainers)
Updated December 7, 2016
OK, here’s the proposed design. @nunit/core-team @nunit/framework-team Please comment.
We’ll define an attribute NUnit.Framework.NonTestAssemblyAttribute. Folks who create extensions that reference the nunit framework, but which are not supposed to be loaded as tests, can use this so that their users don’t have to worry about the problem.
Modify RuntimeFrameworkService.ApplyImageData to notice this attribute and add setting to the test package - something like property IsNonTestAssembly = true.
We’ll define a command-line option for the console runner
--skipNonTestAssemblies(or some other name) that allows users to indicate that there may be some extraneous assemblies on the command line. The runner will use this to set SkipNonTestAssemblies to true in the package.When getting assemblies from a Visual Studio solution, we should set SkipNonTestAssemblies to true for that package.
When getting runners for packages, we’ll skip any that have the setting NonTestAssembly set to true.
When getting drivers for packages, if SkipNonTestAssembly is set we will not return an error should a driver not be found.
Questions? Comments? Does this make sense?
@CharliePoole Unfortunately, we have no separate test assemblies. We use Teamcity for building our software and with NUnit 2 we just ran the tests for all dlls. Since not all assemblies contain tests, it fails with NUnit 3. My suggestion would help people with similar issues migrate to NUnit 3.
We hope to release 3.6 late next week
I’m also not a fan of requiring an NUnit-specific attribute in non-nunit assemblies. I agree there’s a number of different problems, and I wonder if implementing two options would be best - for different use cases.
A
NonTestAssemblyAttribute- for use primarily by libraries which reference the framework but contain no tests, e.g. @ig-sinicyn’s library. This would allow library authors to define the ‘no-tests’ property at their end, without their end users needing to care. The attribute could of course also be used by any users who wished to mark all non-NUnit-test assemblies individually, rather than relying on a global ignore all.A
--ignoreNonTestAssemblyconsole runner flag. (With a better name!) This allows the user running the tests to also have control over this behaviour, and is a preferable option for e.g. TC glob’s or VS solution running, where the user can declare that they are expecting non-test assemblies in this configuration.In our case a simple “ignore all assemblies containing no tests” command line argument would be the simplest approach. We have don’t have separate test assemblies, so any assembly could contain tests. And even if it currently contains no tests, it could tomorrow. Therefore, with NUnit 2, we tested all assemblies.
Having a special attribute would mean that the person adding tests for the first time has to remove this attribute. And a person removing tests (e.g. after refactoring) would have to check if there are still tests in this assembly.
Having the command line argument would mean he doesn’t have to worry about that. We simply would add the command line argument in our build server and done.
@ChrisMaddock That’s what I’m getting out of this as well. As a refinement, we could do it automatically when expanding a VS Solution. I think we already have an issue for that.
I’ll go back to my “design” comment and update it.