nunit3-vs-adapter: Incorrect format for TestCaseFilter Error: Missing '('

Environment

NUnit 3.12
NUnit Adapter 3.16
VS Professional 2019 Version 16.4.2
.NET Framework 4.7.2

Description When executing a category of tests (e.g. Run All), if a test is included in the list of tests that contains an escaped double quote followed by a closing parenthesis, the following error may occur and the test run will fail:

An exception occurred while invoking executor ‘executor://nunit3testexecutor/’: Incorrect format for TestCaseFilter Error: Missing ‘(’.

Note There are a number of recently closed issues (thanks 😃 ) that are similar regarding braces or quotes in the test case name or arguments, but this is not quite the same (esp. since it breaks despite resolution of the other issues)

Reproduction Run this code, and examine the Test Output window to see the exception as per above.

    public class Tests
    {
        [TestCase("Works fine", ")")]
        [TestCase("Works fine", "()")]
        [TestCase("Works fine", "(")]
        [TestCase("Works fine", "\"(")]
        [TestCase("Works fine", ")\"")]
        [TestCase("Works fine", "(\"")]
        [TestCase("Works fine", "()\"")]
        [TestCase("Breaks test executor when using Run All", "\"()")]
        [TestCase("Breaks test executor when using Run All", "\")")]        
        public void TestName(string expectedBehaviour, string data)
        {
            //Irrelevant
        }
    }

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 16
Comments: 33 (10 by maintainers)

Commits related to this issue

repro for https://github.com/nunit/nunit3-vs-adapter/issues/691 — committed to mikhail-barg/jsonata.net.native by mikhail-barg 3 years ago
repro for https://github.com/nunit/nunit3-vs-adapter/issues/691 — committed to mikhail-barg/jsonata.net.native by mikhail-barg 3 years ago
Tests are crashing due to NUnit bug: https://github.com/nunit/nunit3-vs-adapter/issues/691 Nunit bug is that InlineData in strings need to have balanced quotes. Update test case. This shouldn't lose c... — committed to microsoft/Power-Fx by deleted user 2 years ago

Most upvoted comments

A workaround for fellow developers / testers who just want to make the tests run again: Wrap your test case data in a TestCaseData object, and return with an IEnumerable<TestCaseData> instead of an IEnumerable<object[]>. Fill the TestName of each of these cases; avoid special characters, and “tadaa” the tests will run again properly.

// Still looking forward to a proper fix though.

igabesz on Jun 18, 2020

I’m getting this error with the latest packages (for vs2019) running in vs 2022:

========== Starting test run ==========
Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
NUnit Adapter 4.1.0.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscoveryContext.GetTestCaseFilter(IEnumerable`1 supportedProperties, Func`2 propertyProvider)
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_TfsTestCaseFilterExpression() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 100
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_IsEmpty() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 102
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.InitializeForExecution(IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 248
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 105
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 131 ms ==========

I’ve created a gist with the relevant files I can think of: https://gist.github.com/redwyre/0fb0c9f177af278f76aa8cf49c1eb1dc

It seems like NUnit is just calling ToString() on all the parameters to make the test name, and in some cases it doesn’t work properly with more complex objects.

I think I can work around this by wrapping each record in TestCaseData and name them, but it overrides the full name of the test not just the postfix. I’m not very experienced with NUnit myself so not sure if there is a better way.

redwyre on Nov 14, 2021

Thanks! We’re aware of these cases too, but thanks for the test code!
There are even more bizarre behavior here. I added some more with explanations at https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue691/Issue691

Part of this is a Visual Studio issue, part of it is the name handling in the adapter - and these things are not properly aligned yet.

OsirisTerje on Jan 9, 2020

@mikhail-barg Thanks for checking this out 😃

OsirisTerje on Oct 10, 2021

Updated all packages in Issue691.sln to latest versions, and still everything work:

========== Starting test discovery ==========
NUnit Adapter 4.0.0.0: Test discovery starting
NUnit Adapter 4.0.0.0: Test discovery complete
========== Test discovery finished: 9 Tests found in 531 ms ==========
========== Starting test run ==========
NUnit Adapter 4.0.0.0: Test execution started
Running selected tests in d:\Projects\nunit3-vs-adapter.issues\Issue691\Issue691\Issue691\bin\Debug\netcoreapp3.1\Issue691.dll
   NUnit3TestExecutor discovered 9 of 9 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.0.0.0: Test execution complete
========== Test run finished: 9 Tests (9 Passed, 0 Failed, 0 Skipped) run in 240 ms ==========

mikhail-barg on Oct 10, 2021

Yes, I think so. Will double-check and try to create a repro

mikhail-barg on Oct 7, 2021

Same issue happening with nunit test adapter v 3.17.0.0.

davidcatriel on Feb 5, 2021

Thanks for the response & update.

I can confirm that I’m also getting test explorer issues as described in issue 691, as well as general issues with test explorer being glitchy and not updating status from the nunit test run.

This is why I cringe every time I update something that works 😢 (VS in this case)

RabidArts on Jan 10, 2020