testfx: MSTest 2.2.4 does not work with [DataTestMethod] [DynamicData] and a complex object

Description

Everything was working fine in 2.2.3. Then we updated to 2.2.4 and it does not work anymore. So we have some classes with [DataTestMethod] and some dynamic data that we feed into it. The tests were green in 2.2.3 and upgrading to 2.2.4 break them with this error:

C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug>dotnet test MsTest224DataTestBug.csproj
 Determining projects to restore...
 Restored C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\MsTest224DataTestBug.csproj (in 529 ms).
 MsTest224DataTestBug -> C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\bin\Debug\net5.0\MsTest224DataTestBug.dll
Test run for C:\Users\mme\Documents\GitHub\MsTest224DataTestBug\MsTest224DataTestBug\bin\Debug\net5.0\MsTest224DataTestBug.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
An exception occurred while invoking executor 'executor://mstestadapter/v2': Type 'MsTest224DataTestBug.MyStringContainer' with data contract name 'MyStringContainer:http://schemas.datacontract.org/2004/07/MsTest224DataTestBug' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
Stack trace:
  at System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeAndVerifyType(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, Boolean verifyKnownType, RuntimeTypeHandle declaredTypeHandle, Type declaredType)
  at System.Runtime.Serialization.Json.XmlObjectSerializerWriteContextComplexJson.SerializeWithXsiTypeAtTopLevel(DataContract dataContract, XmlWriterDelegator xmlWriter, Object obj, RuntimeTypeHandle originalDeclaredTypeHandle, Type graphType)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObject(XmlWriterDelegator writer, Object graph)
  at System.Runtime.Serialization.XmlObjectSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
  at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(XmlDictionaryWriter writer, Object graph)
  at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(Stream stream, Object graph)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.DataSerializationHelper.Serialize(Object[] data)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ToTestCase()
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.SendTestCases(String source, IEnumerable`1 testElements, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext, IMessageLogger logger)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.DiscoverTestsInSource(String source, IMessageLogger logger, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
  at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
  at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
  at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)


Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

Steps to reproduce

Here is a full sample reproducing the bug: https://github.com/TheFireCookie/MsTest224DataTestBug In this repo, there is a folder MsTest223DataTestBug in which the command ‘dotnet test’ works correctly, the other folder is MsTest224DataTestBug and in this one, it’s exactly the same code except I’ve upgraded MSTest to 2.2.4 and there a ‘dotnet test’ fails with the error written earlier.

Expected behavior

No regression and we can still use [DynamicData] with complex objects

Actual behavior

We cannot use anymore [DynamicData] with complex objects, only simple types like string, int etc

Environment

Tested and reproduced on Windows 10 and CentOS 7.9. It’s running on .NET 5.0.

AB#1334620

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 28
Comments: 19 (9 by maintainers)

Commits related to this issue

Création d'un projet de reproduction du problème (https://github.com/microsoft/testfx/issues/844) — committed to ybru-isiware/MsTest224DataTestBug by deleted user 3 years ago
Fixed a bug in `ITestDataSource` data deserialization (#864) - Fixed a bug in `ITestDataSource` data deserialization when trying to deserialize a type outside of current assembly. This should fix th... — committed to microsoft/testfx by Haplois 3 years ago
Squashed commit of the following: commit f7de2d4fdeccbb318cdebeb23f0b488e5ac570b8 Author: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com> Date: Wed Jul 21 14:26:59 2021 ... — committed to HeroMaxPower/testfx by HeroMaxPower 3 years ago

Most upvoted comments

Thank you for the report. This bug is caused because of a change in how we handle attributes derived from ITestDataSource.

I am working on a fix, and will release a preview Monday for this bug.

+12

Haplois on May 27, 2021

@Haplois

With #864, last failing case is fixed too. We also added some tests.

Is there a timeline on when 2.2.5 is planned to be released?

japj on Jun 16, 2021

This is hopefully fixed, and a preview is available in our NuGet feed.

Fixed packages:

Could you please test it and verify if this fixed the issue?

/cc: @TheFireCookie, @sk6868, @DOliana, @davesmits, @rikrak, @andessi, @Akinzekeel, @chriswelles, @conficient, @xantiva

Haplois on Jun 4, 2021

@Haplois thanks for the quick update.

I just tested the preview and re-ran all of my tests. The ones that were bugged out before (using types such as double and char) are working now. However a new bug appears to have popped up.

The following kind of tests are failing for me now (they worked previously in 2.2.3, not sure about 2.2.4):

[DataTestMethod]
[DataRow("john.doe@example.com", "abc123", null)]
[DataRow("john.doe@example.com", "abc123", "/unit/test")]
public async Task Can_SignIn(string Email, string Password, string ReturnUrl) { ... }

This is the error reported in the test explorer for these tests:

  Message: 
    Value cannot be null. (Parameter 'typeName')

  Stack trace: 
    RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
    Type.GetType(String typeName)

Update: there is now also an issue with byte-based enums (int-based enums work)

Message: 
    Test method My_Test_Method threw exception: 
    System.ArgumentException: Object of type 'System.Int32' cannot be converted to type 'MyByteBasedEnum'.

  Stack trace: 
    RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
    RuntimeType.CheckValue(Object value, Binder binder, CultureInfo culture, BindingFlags invokeAttr)
    MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
    RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    MethodBase.Invoke(Object obj, Object[] parameters)
    ThreadOperations.ExecuteWithAbortSafety(Action action)

Akinzekeel on May 31, 2021

Thank you @ybru-isiware,

I was able to repro the issue. Working on it.

Haplois on Jun 4, 2021

When testing update to 2.2.4 I ran into an issue with string[] and int[] params in DataRow attributes causing test-discovery exception. Developer Community ticket someone posted here: https://developercommunity.visualstudio.com/t/cant-set-arrayofstring-as-the-datarow-in-mstest/1435155

Also I had an issue with floating point value params (1.5, 1.5f, 1.5d) being casted to decimal value and causing test failures.

I’ve run our tests against v2.2.5-preview-20210604-01 posted above - both of the issues I ran into in 2.2.4 seem to be resolved.

PRRB on Jun 4, 2021

So my test with the complex objects are working now.

I’ve also tested the method of @Akinzekeel and it seems to work fine too.

When can we expect a real release of MSTest with the fixed issues? Have a nice day!

TheFireCookie on Jun 4, 2021