nunit: Failing Unit Tests in nunit.framework.tests After New Clone
I just grabbed a fresh clone of nunit and popped open the solution in Visual Studio 2017 (Community 15.7.4) with the latest NUnit 3 Test Adapter (3.10 with Run Tests in Parallel) running on Windows 10 Professional and selected The “Debug-AnyCPU” and hit build.
After reading the BUILDING.md it seems like I should be ignoring any failures that don’t come from nunit.framework.tests-* and nunitlite.tests-*
However out of the gate I get 2 failures from nunit.framework.tests-*
First Error:
Test Name: TestCaseSourceCanAccessWorkDirectory("C:\\Users\\ace.olszowka\\source\\nunit\\bin\\Debug\\net20")
Test FullName: NUnit.Framework.TestContextTests.TestCaseSourceCanAccessWorkDirectory("C:\\Users\\ace.olszowka\\source\\nunit\\bin\\Debug\\net20")
Test Source: C:\Users\ace.olszowka\source\nunit\src\NUnitFramework\tests\TestContextTests.cs : line 110
Test Outcome: Failed
Test Duration: 0:00:00.001
Result StackTrace: at NUnit.Framework.TestContextTests.TestCaseSourceCanAccessWorkDirectory(String workDirectory) in C:\Users\ace.olszowka\source\nunit\src\NUnitFramework\tests\TestContextTests.cs:line 112
Result Message:
Expected string length 34 but was 50. Strings differ at index 34.
Expected: "C:\\Users\\ace.olszowka\\source\\nunit"
But was: "C:\\Users\\ace.olszowka\\source\\nunit\\bin\\Debug\\net20"
-------------------------------------------------^
Looking at the source I don’t see how this was possible, as far as I can tell these values should have been identical (_workDirectory and the test data are both set to TestContext.CurrentContext.WorkDirectory) my only guess is some type of race condition, maybe due to bad configuration on my end?
Second Error:
Test Name: StackTracesAreFiltered("WarningInBeginInvoke",4)
Test FullName: NUnit.Framework.Assertions.WarningTests.StackTracesAreFiltered("WarningInBeginInvoke",4)
Test Source: C:\Users\ace.olszowka\source\nunit\src\NUnitFramework\tests\Assertions\WarningTests.cs : line 292
Test Outcome: Failed
Test Duration: 0:00:00.004
Result StackTrace: at NUnit.Framework.Assertions.WarningTests.StackTracesAreFiltered(String methodName, Int32 maxLineCount) in C:\Users\ace.olszowka\source\nunit\src\NUnitFramework\tests\Assertions\WarningTests.cs:line 310
Result Message:
Multiple failures or warnings in test:
1) (Warning message)
2) Expected the number of lines to be no more than 4, but it was 5:
1. at NUnit.TestData.WarningFixture.<>c__DisplayClass45_0.<WarningInBeginInvoke>b__0()
2. at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
3. at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
4. at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o)
5. at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
(end)
I get lost in what this test is attempting to do here; but based on my limited understanding it seems like this code is really sensitive towards any changes in the call stack, could this be due to a recent change in the .NET Framework?
I also get every single test case failing in NUnitLite.Tests.CommandLineTests I would be happy to dig on there if this is unexpected.
Seeing as the build is passing in CI these are probably just configuration issues on my end, but nothing was mentioned in BUILDING.md about this so figured a report was worth it.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 31 (29 by maintainers)
No, thanks for pointing it out. Closing.
Just trying to do house cleaning; is there any reason that this issue should still be open or can we close it with some type of resolution? (Even if it is WONTFIX).
Works for me. My CF build passes 100% of the NUnit tests.
@jnm2 I like your “canonical” approach. I’m dubious that it will be easy to make everything work smoothly in Test Explorer, but it’s worth trying.
It’s still useful, it essential, for anyone working on nunit to know when to switch to a lower level of testing.
Biggest thing we could do to move this idea forward IMO is to establish separate system and maybe integration tests.
In the past, I just told people to not use the test explorer for the NUnit tests. Other team members have pushed back against that saying that test explorer has become, for many people, the default way to run tests.
There is certainly ample precedent for a particular runner not being useful when testing a testing framework. Testing frameworks under test are a pretty special situation and not one encountered by most users. OTOH, if the team wishes, maybe you will make it a goal to be able to successfully run under test explorer through the adapter.
Whatever you do, I think you should probably put out some documentation that tells people that running NUnit tests under the test explorer does not currently work. You can tell them in that context either that it’s a goal to make it work or that it’s not a priority, whichever is decided.
As the first author of the adapter and a many-year contributor to the framework, I have always decried the use of the test explorer in the development of NUnit. That’s still my view on the issue. Heck, I don’t even like that we use the console to test the framework in our CI now!!!
Whatever the decision, I think you are going to have to keep sending people back to the actual NUnit runners (either NUnitLite or the console runner) when issues come up that may or may not be due to the adapter. The adapter remains fundamentally equivalent to a third-party runner, even if it’s under the NUnit Project.
Thanks for the report! I’ve confirmed that
TestCaseSourceCanAccessWorkDirectoryandCommandLineTestsdon’t work in Test Explorer. We need to come up with a way to make them independent of runner and parallelism. @nunit/framework-team Any ideas?For the second error, that’s a smell test. It makes sense to bump that to 5.