roslyn-sdk: xUnit.net support broken with xUnit.net 2.5

In upgrading the xUnit.net analyzers project to use a prerelease 2.5.0 build, I’ve determined that Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.XUnit (and/or some of its dependencies) are broken when my code uses 2.5.

Any verifier which fails breaks because of a breaking binary dependency on several of the xUnit.net exception classes. (The assertion library, including all the exceptions, has been overhauled for v2 2.5 and v3.) The source of the problem is that you’re taking a binary dependency on xunit.assert, which is locking consumers into your version choice when breaking changes are made to anything you consume.

This behavior isn’t appropriate, in my view. Your dependency in essence locks users down to only being able to use “compatible” versions of xUnit.net. This is true for any binary dependency you take outside of the core framework: if you ship a hard dependency on a binary that someone else may want to use in their tests, then you are potentially locking them down to only being able to use a version that’s compatible with your choice. I ended up removing all binary dependencies from xUnit.net v2 for exactly this reason; for us, the problematic library ended up being JSON.NET, and it broke JSON.NET when they wanted to test themselves with us. Even if we’re one of the few or only consumers of this library, we’re guaranteed to want to “move on” to newer versions of xUnit.net. 😄

Here is a quick example showing the problem. Given the follow code which should clearly fail:

This is the (expected) result using xUnit.net 2.4.2:

And this is the (incorrect) result using xUnit.net 2.5.0-pre.22:

I can think of at least two suggestions for potential paths forward.

1. Don’t use xunit.assert at all.

xUnit.net considers all exceptions to be failing tests. Don’t consume the xUnit.net assertion library; instead, create your own exceptions with your own messages and throw those.

2. Use xunit.assert.source (or Git sub-module) instead.

If you wish to consume the library but not ship a binary dependency on it, you can import the source version of our library and use that instead. The project is designed to be consumed in both of these fashions.

By default, importing the source (via NuGet or sub-module) will bring in all the code as public, but you can override that with XUNIT_VISIBILITY_INTERNAL. You can use pre-processor directives to influence which features are enabled or disabled. By leaving everything as internal, you will be able to use the existing assertions without republishing them into your API.