nunit3-vs-adapter: NUnit3TestAdapter 3.15 not running tests with custom TestCaseSource

NUnit 3.12.0 NUnit3TestAdapter 3.15.0 Visual Studio 2019 Version 16.1.3 .NET Framework 4.6.1

namespace NUnitTest
{
   public class UnitTest1
    {
        public class DataProvider<T> where T : TestData, new()
        {
            public static IEnumerable<TestCaseData> GetData()
            {
                string[] names = new string[] { "Matthew", "Kelly", "Shawn" };
                char[] genders = new char[] { 'M', 'F', 'M' };
                int[] ages = new int[] { 35, 34, 30 };

                for (int i = 0; i < names.Length; i++)
                {
                    TestData testData = new TestData()
                    {
                        Name = names[i],
                        Gender = genders[i],
                        Age = ages[i]
                    };

                    TestCaseData tcd = new TestCaseData(testData);
                    tcd.SetName("{m} [" + testData.Name + " - " + testData.Gender + " - " + testData.Age + "]");

                    yield return tcd;
                }
            }
        }


        [Test]
        [TestCaseSource(typeof(DataProvider<TestData>), "GetData")]
        public void Persons(TestData test)
        {
            Assert.IsTrue(test.Name == "Matthew");
        }
    }
}

The above code correctly generates 3 test cases that have the following names in the Test Explorer:

Persons [Kelly - F - 34]
Persons [Matthew - M - 35]
Persons [Shawn - M - 30]

However, when I attempt to run/debug any of those test cases, they disappear from the Test Explorer, and nothing happens. A simple rebuild (or another clean and build) brings them back, but they still cannot be executed as they just disappear again.

Using the debugger, the custom data

If I downgrade to NUnit3TestAdapater 3.14, everything works as expected, I get 2 failed and 1 passed.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (11 by maintainers)

Most upvoted comments

Thanks @MattKeenum ! Based on this case, in addition to the other, I am pretty sure we need to rollback 3.15. That performance work is unfortunately not finished.

OsirisTerje on Aug 27, 2019

Edit: Just to be clear, I am only talking about how the Test Explorer displays the names of the tests. All of the other functionality works great.

@jnm2 Sure. Our company has over 500 applications that all need automated testing. Since we are mostly a C# shop, we chose NUnit as the testing framework of choice. As each team developed their own framework to write automated tests, we realized that every team was solving the same problems in different ways.

We wrote a generic wrapper around NUnit and distribute it as a NuGet package. This lets us solve the common problems once, and easily allows team members to move from team to team without having to learn a different framework.

Apart of this, we heavily customize the TestCaseSource to be able to create tests from different data sources, like Excel, CSV, Databases, etc. We then duplicate these tests changing slight configurations, like which environment to test, where the tests are executed (Local, Selenium Grid, Sauce Labs, etc.). If the team is using mobile devices, it allows the tests to be duplicated for different devices. This allows us to build once and run the tests we need to when approaching the CI/CD pipeline.

Unfortunately, I am prohibited by my company to upload pictures, but here are some example test names:

[Local : Dev : Win : Chrome] - [Card] - [MyMethod1] - [MyDataSourceTestName] [Grid : Dev : Win : Chrome] - [Card] - [MyMethod1] - [MyDataSourceTestName] [Sauce Labs : QA : Mac : Safari] - [Card] - [MyMethod2] - [MyDataSourceTestName]

Having full control over how the names are presented, makes it much easier to look through the Test Explorer for the tests the end user wants to run.

I’m open to ideas. We aren’t 100% sold on SetNames. Just realize that any changes we make to the way the Test Names are displayed in the Test Explorer, affects 100s of people using the framework. So we want to be consistent in how we display these names to reduce confusion among the teams.

MattKeenum on Aug 29, 2019

@MattKeenum I’m not sure exactly what’s happening with SetName. I think it might be losing the namespace information from the test names. We will keep looking into it, but in the meantime, NUnit Framework has an API that doesn’t break in this situation which is IMO a superior way of setting test names. (It doesn’t have funky problems like unescapable tokens getting replaced the way SetName does.)

Instead of:

tcd.SetName("{m} [" + testData.Name + " - " + testData.Gender + " - " + testData.Age + "]");

See what you think of:

tcd.SetArgDisplayNames(testData.Name, testData.Gender.ToString(), testData.Age.ToString());

jnm2 on Aug 28, 2019

Be aware that the vsix extension is being deprecated. It is advised that you use NuGet to install NUnit. http://hermit.no/vsix-based-test-adapters-to-be-deprecated-in-visual-studio-for-c-and-visual-basic-to-speed-up-testing/

MattKeenum on Aug 28, 2019

@OsirisTerje I can confirm that the fix is working for 3.15.1. Thanks everyone!

MattKeenum on Aug 30, 2019

@MattKeenum 3.15.1 hotfix released now.

OsirisTerje on Aug 30, 2019

@MattKeenum That makes sense. Thanks for the detail. SetArgDisplayNames is by design not the way to go for your use case. We’ll continue to look at what it’s going to take to get SetNames interacting well with the prefilter.

jnm2 on Aug 29, 2019

@OsirisTerje I wouldn’t mind if you commented on that thread though 😄

jnm2 on Aug 28, 2019

@turcunicusor You would need to manually remove the reference to the TestAdapter, and then add a reference to the previous version. As a double-check, you can check your App.config, csproj file and packages.config files for any references pointing to the 3.15 version and change them to the 3.14 version.

MattKeenum on Aug 28, 2019