nunit-console: Should --Explore Error On "Invalid" Assemblies?

Reproduce With The Following:

echo.>bad.dll
nunit3-console.exe -explore bad.dll

Yields:

NUnit.Engine.NUnitEngineException : bad.dll could not be examined
  ----> System.BadImageFormatException : Format of the executable (.exe) or library (.dll) is invalid.

--NUnitEngineException
bad.dll could not be examined
   at NUnit.Engine.Internal.TargetFrameworkHelper..ctor(String assemblyPath)
   at NUnit.Engine.Services.RuntimeFrameworkService.ApplyImageData(TestPackage package)
   at NUnit.Engine.Services.RuntimeFrameworkService.ApplyImageData(TestPackage package)
   at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFramework(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.GetEngineRunner()
   at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
   at NUnit.ConsoleRunner.ConsoleRunner.ExploreTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)
--
BadImageFormatException
Format of the executable (.exe) or library (.dll) is invalid.
   at Mono.Cecil.PE.ImageReader.ReadImage()
   at Mono.Cecil.PE.ImageReader.ReadImageFrom(Stream stream)
   at Mono.Cecil.ModuleDefinition.ReadModule(Stream stream, ReaderParameters parameters)
   at Mono.Cecil.ModuleDefinition.ReadModule(String fileName, ReaderParameters parameters)
   at NUnit.Engine.Internal.TargetFrameworkHelper..ctor(String assemblyPath)

Obviously because this is a completely invalid DLL.

Our use case is a little more exotic than this, basically we have a common bin output location in which hundreds of DLL’s end up, some managed, some unmanaged, and we are asking the question “Which of these have tests?”.

The error in question comes from:

https://github.com/nunit/nunit-console/blob/0958996f2735c213bad7e8ed1c81e4d4ffdfec11/src/NUnitEngine/nunit.engine.core/Internal/TargetFrameworkHelper.cs#L44

We have similar code in our code base, however we simply ignore the load failure.

The question is more one of design, should this be the expected behavior? Or should --explore be “Best Effort”?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (12 by maintainers)

Most upvoted comments

Thanks for starting the conversation @aolszowka! #798 now covers the crash, and I’ve just created #827 to cover the potential change in behaviour, which I personally think would be a good one! 🙂

ChrisMaddock on Nov 13, 2020

Yeah, I think we have jumped back and forth between explore creating test input and the user creating a list of assemblies. My thinking on the latter was that the list, generated simply enough on the command-line would go in a response file. The hypothetical --skipxxxx option would tell NUnit “Don’t be surprised if some of these files can’t be loaded.”

I think that’s the problem we started out with here.

CharliePoole on Jul 22, 2020