vstest: dotnet test for multi target net461 and netcoreapp3.1 success in windows but fail in Ubuntu 20.04 with error: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel

@nohwnd Also, If the package Microsoft.VisualStudio.TestPlatform.ObjectModel is added as a reference, test success. The package Microsoft.VisualStudio.TestPlatform.ObjectModel is used by Xunit project and it’s one of the dependencies of Microsoft.TestPlatform.ObjectModel. It provides interfaces such as ITestExecutor and ITestDiscoverer for implementing custom unit test adapters.

It was requested by @bradwilson, the author of Xunit, here

The referenced assembly is a contract assembly which is expected to be provided by the version of Visual Studio (and/or VSTest) which is running the tests.

This package is used also by netcoreapp3.1 but it’s loaded during netcoreapp3.1 build but it’s not loaded during net461 build. This part of .deps.json for netcoreapp3.1 project:

"Microsoft.TestPlatform.ObjectModel/16.6.1": {
        "dependencies": {
          "NuGet.Frameworks": "5.0.0"
        },
        "runtime": {
          "lib/netstandard2.0/Microsoft.TestPlatform.CoreUtilities.dll": {
            "assemblyVersion": "15.0.0.0",
            "fileVersion": "15.0.0.0"
          },
          "lib/netstandard2.0/Microsoft.TestPlatform.PlatformAbstractions.dll": {
            "assemblyVersion": "15.0.0.0",
            "fileVersion": "15.0.0.0"
          },
          "lib/netstandard2.0/Microsoft.VisualStudio.TestPlatform.ObjectModel.dll": {
            "assemblyVersion": "15.0.0.0",
            "fileVersion": "15.0.0.0"
          }
        },

I noticed that the package has no project reference and it’s not maintained since 2017, but the package is included in netcoreapp3.1 as version 16.6.1.

Suggested solution The package Microsoft.TestPlatform.ObjectModel should be part of the dependencies of the package Microsoft.NET.Test.Sdk in net4x and it can solve many like issues in Linux/Mac with Mono installed.

I shipped 2.5.3 yesterday with the NuGet package dependency against Microsoft.TestPlatform.ObjectModel and we’ll see how users get along with that. I suspect the number of users trying to run xUnit v2 tests on Mono (which is not officially supported) is relatively low, and probably the majority of them had already fixed this with a manual package reference.

If that blows up in unexpected ways, I’ll consider pulling it back out and replacing it with this assembly resolver. We already do assembly resolution for the test adapter path, which I suspect was/is only necessary inside Visual Studio (as dotnet test and friends currently depend on us copying the test adapter into the build output folder).

After mulling it over last night, and doing some testing w/rt to version mismatches (no issues I could uncover), I’ve decided to add the dependency to Microsoft.TestPlatform.ObjectModel. Whether this issue gets fixed here or not (which it absolutely should, IMO, since it has affected every unit test framework and not just xUnit.net), we cannot assume that users are diligently updating their reference to Microsoft.NET.Test.Sdk to grab the latest version. This reference is a “chore” reference that’s left to the user to add in order to make unit testing work at all, and not something that developers interact with directly.

https://xunit.net/releases/visualstudio/2.5.2-pre.10

All that said? Please, still, fix this bug.