testfx: Test Explorer fails to show tests. Tests output: "The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)"

I generated the fusion logs - I had a quick look but I couldn’t see any errors.

I do have another test project that works fine and its tests do appear in the Test Explorer window - so I compared the two projects. I noticed they’re both .NET Framework 4.6.2 projects, however the working project has a reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework which is being pulled from C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll - whereas the non-working project has a reference to Microsoft.VisualStudio.TestPlatform.TestFramework which is being pulled from (solution root)\packages\MSTest.TestFramework.1.1.18\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll.

Indeed, the working project is using a standard Visual Studio Test project which came with its own VS-coupled references, while the non-working project is using the NuGet “MSTest V2” assemblies instead.

So why is MSTest V2 not working all of a sudden?

TL;DR Workaround:

1. Remove the NuGet package references and project assembly references to Microsoft.VisualStudio.TestPlatform.TestFramework.
2. Replace them with a reference to your Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll assembly instead, which is located inside your Visual Studio installation directory (Common7\IDE\PublicAssemblies).

Just want to report that I had the same problems in a newly created project. The app.config used to be empty, but after installing some packages assemblybinding’s must have gotten added, which was causing the problem.

The solution was simple - I deleted the whole runtime section and cleared the output folders (bin, obj and TestResults), then it was working again. Tried adding the assemblybindings again and got the same error.

Update your MSTest.TestAdapter to 1.2.1 and MSTest.TestFramework to 1.2.1.

These packages will be available in nuget.org very soon. Until then you can obtain them from feed https://dotnet.myget.org/F/mstestv2/api/v3/index.json

After updating you can see message with exact missing assembly name.

I’m closing this issue as there is no more action required from testfx side.

Microsoft should just get rid of their forum websites and use Stackoverflow completely. Every time I have a problem and I do a web search and end up at Microsoft’s site, I find completely useless information. Usually, it is from a Microsoft employee who clearly is a newbie who doesn’t have a clue. It is just a waste of time even looking at that site.

I found the reason for the failure on my end.

Basically some of my projects are really old, and have slowly been migrated to newer .NET-versions and test-frameworks.

Turns out that the app.config in my project I had the following section still having around, despite the project being “upgraded” to .NET 4.5.1 and latest MSTest:

  <system.codedom>
    <compilers>
      <compiler language="c#;cs;csharp" extension=".cs" type="Microsoft.CSharp.CSharpCodeProvider, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" warningLevel="4">
        <providerOption name="CompilerVersion" value="v3.5"/>
        <providerOption name="WarnAsError" value="false"/>
      </compiler>
    </compilers>
  </system.codedom>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1" appliesTo="v2.0.50727">
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Extensions" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Extensions.Design" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0"/>
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

Removing these made test-explorer able to discover and run my tests.

@cltshivash

1. We have considered a solution level migration. We plan on supporting it in future. You can track it at https://github.com/NuGet/Home/issues/5879.
2. We have a plan of supporting PR on ASP.NET full framework, but there are some open questions that are still need to be sketched out. You can follow the larger effort to improve the PC to PR experience at https://github.com/NuGet/Home/issues/5877.

I’m not sure I care if its nuget or VS really. Its bugs are creating more work for me in addition to creating this nasty bug. I just want them fixed without catching a bunch of flak.

If you are facing Nuget issue always share repro project and raise issue at https://github.com/NuGet/NuGet.Client.

@smadala - the reported issue is not a Nuget issue and I’m not sending whole 50-200 project solutions over when they seem to be too busy tinkering with low value things (and breaking stuff along the way) and placation to be bothered with long awaiting bugfixes.

I’ve only had to log maybe 5 issues in the last 20 years for MSFT. But the amount of time I’ve had to spend to work around or otherwise deal with recent MSFT quality issues (230+) has totaled about two months for this year. Just this one issue alone has cost me about thee work days, not counting the effect of bugs in software that weren’t caught by the unit tests because they were silently skipped during gated checkin builds (they mysteriously stopped working earlier this year - another multi day issue).

If you all could stop screwing around with all the shiny things (nodejs, git, and others your established .net customer base does not care about), and focus on improving the quality of the things that are foundational, I may just get my nights and weekends back in 2018.

Getting additional logs is just more time I lose, and I’m not sure what (if anything) you could gain except more time to service other issues until then. The above information I shared should be enough to repro this. You guys probably need to try it out on a non-MSFT image workstation or other test environments.

@mishra14 Do you have recommendation for the below ?

• Ability to migrate at solution level
• Timeline on when would Package reference be available for Asp.net full framework

@cltshivash - A tool is for sure better than the semi-manual steps. I’d still have to click on it for every single project (10 - 180 per solution) unless I can action it at the solution level.

However, the first three of these, in addition to the “Not yet available for Asp.net full framework” are complete blockers for us. Thanks for the info share though.

@cltshivash - The nuget team owes a conversion/migration tool (its close to done I think) before we can realistically convert to package ref. If the problem with the silent failure to discover tests gets fixed I can at least not worry about that part of it. I’d rather have it break on build so we can workaround it sooner.

I offer you the advice of not prematurely closing this thread. When the package is available and the OP or participants in this thread deem it fixed satisfactorily, that would be the time to close it.

+1

Well put @StingyJack ⚡!

After updating you can see message with exact missing assembly name.

@smadala - getting the assembly name is only identifying the assembly that is missing. The STR and E/A complain about tests not being discovered, not that we don’t see the assembly name its complaining about.

I’m closing this issue as there is no more action required from testfx side.

If the assembly that is missing (I presume you know what that assembly is at this point) is a MSFT assembly, then there is still action required from the MSFT side.

As someone who has too many years in the same kind of customer service role you are holding in within this thread, I offer you the advice of not prematurely closing this thread. When the package is available and the OP or participants in this thread deem it fixed satisfactorily, that would be the time to close it. @ManishJayaswal attests that there are no issue closure stats being tracked at MSFT for measuring individual performance, so leaving this open until that agreement happens should not pose much of a problem, if any.

@StingyJack - I can categorically state that closure stats is not even a consideration in the company. I am mentioning this since you have brought this up several times.

In this specific case - as I have mentioned to @spottedmahn above, please try the simple workaround of removing the binding redirect. TestPlatform ideally should take dependency on the minimum possible version of System.Runtime assembly so we are working with the TestPlatform team to see what can be done here. Hence we are keeping the issue open. However please let us know if the workaround does not work for you and if this issue is a blocking issue for you.

As a workaround for now, you can comment out the binding redirect ( which I believe you are already doing). Test will use newer System.Runtime and it should be fine. We would need to investigate to see if we can move TestPlatform dependency to a lower version but I don’t expect that change to happen quickly. Thanks for helping us get to the root of this issue. Much appreciated @spottedmahn !!

@spottedmahn - you are right the execution does not work and removing the binding redirect makes the test run. This indicates that TestPlatform has a dependency on a newer version of System.Runtime. I will follow up with TestPlatform team to understand this better but can you tell us why you need to depend on an earlier version of System.Runtime?

@singhsarab FYI

@spottedmahn I tried your repro on the latest 15.6 preview bits ( which just got released - like 15 minutes ago- so you can download and try it too). When I opened the solution, I noticed that the reference to System.Runtime ( blue circle in the image below in solution explorer window - kind of hard to see, sorry about my poor choice of color) was not resolved ( had a yellow triangle on it). However it got resolved when I built the solution and then the tests were getting discovered. Real time test discovery was also working ( I verified it by adding a removing a new test and TE window was getting update in real time). Please give the latest preview a try and let us know if this is still an issue.

Ah sorry - I was using this repro provided by @StingyJack - (https://github.com/StingyJack/testfx241). We will try the other one too and post back our findings. @spottedmahn - Thanks for pointing this out!

Per the title, it is the same issue. the thread is only getting too long as its not been resolved and more people are submitting claims and evidence.

Its not too unweildy for me, and the dev comm site is frustrating in so many ways (cant search through own issues, formatting is ne’er impossible, notifications, mods can lock out OP replies, etc), why prefer it?

@StingyJack Thanks for bring that up, As this thread getting too long, We will track the issue separately. Please create another issue with repro steps and logs(/diag:log.txt).

From MSTest testadapter side, Error message need to improve. I will keep this issue open to address it.

@smadala - there is also the issue of it continuing to execute and “skip” tests without any failure indication or exception. It did this for months on the build servers before I noticed it happening locally

@spottedmahn Thanks for repro. It seems like core issue (Fail to load assembly(System.Runtime)) not related to testadapter.

Create console app similar to test project you created and add Assembly.Load("System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"); causes same failure as test discovery/run. Copying System.Runtime.dll from ClassLibrary’s OutputDirectory to ClassLibrary.Tests/ConsoleApp OutputDirectory fixes the assembly load.

In meeting, We thought System.Runtime.GCSettings.IsServerGC from System.Runtime assembly, Actually it is from mscorlib.dll.

I’m not sure whose(NuGet.Client/ msbuild) issue is this? Can you create issue on them?

From MSTest testadapter side, Error message need to improve. I will keep this issue open to address it.

@spottedmahn - long thread is long

Hi @smadala - I’ve managed to reproduce it in a sample app 🎉

Please see this project and readme.md for details.

@spottedmahn Sent you mail for meeting.

@StingyJack Default template won’t create app.config, I’m not sure about nuget or any other tool.

@jemiller0 During tests discovery/run test-adapter(MSTest v2) loads test-assembly(UnitTests.dll) in new appdomain(AD2) with test-assembly config(UnitTests.dll.config) as application configuration file for AD2. In AD2 all the binding redirects(doc) and probing paths(E.g: <probing privatePath="bin;bin2\subbin;bin3"/>) in test-assembly config applied for any assembly loading. That is why assembly load failed if config file has redirects to assembly version which is not present in probing paths. See @spottedmahn shared logs “binding errors\Default\testhost.exe\System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.html” for assembly resolving process.

• Why we need to create new appdomain and apply application configuration? To create isolated runtime for tests.E.g
  Test discovers/runs in a process (E.g: testhost.exe) has dependency on Newtonsoft.Json: 9.0.1 but test assembly has dependency on 10.0.3.

@spottedmahn Is your error code is 0x80070002? Assembly resolving failing for “System.Runtime”. C:\Users\mdepouw\Source\Repos\MyProject\MyProject.Library\bin\Debug\MyProject.Library.dll.config has binding redirects for System.Runtime 4.0.0.0 to 4.1.1.0? Look at “System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.html and System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.html”.

LOG: Redirect found in application configuration file: 4.0.0.0 redirected to 4.1.1.0.
LOG: Post-policy reference: System.Runtime, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Remove redirects for System.Runtime or make sure it present in GAC/probing paths.

Can you share steps for how did you get into this problem?

Commenting out the bindingRedirects in my test project and then regenerating them seems to have fixed the problem for me. For the time being anyway. The following is what I currently have. The commented out ones are what it sued to be. I used Get-Project -Name OleLibraryTest | Add-BindingRedirect to regenerate the binding redirects.

  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <!--<dependentAssembly>
        <assemblyIdentity name="System.Threading.Tasks.Extensions" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-10.0.0.0" newVersion="10.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Reflection" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime.InteropServices" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Buffers" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime.Extensions" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Text.RegularExpressions" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.1.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore.Relational" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="MySqlConnector" publicKeyToken="d33d3e53aa5f8c92" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-0.33.1.0" newVersion="0.33.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Npgsql" publicKeyToken="5d8b90d52f46fda7" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-3.2.6.0" newVersion="3.2.6.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />
        <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" />
      </dependentAssembly>-->
      <dependentAssembly>
        <assemblyIdentity name="System.Threading.Tasks.Extensions" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Buffers" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.0.2.0" newVersion="4.0.2.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.EntityFrameworkCore.Relational" publicKeyToken="adb9793829ddae60" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.1.0" newVersion="2.0.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Npgsql" publicKeyToken="5d8b90d52f46fda7" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-3.2.6.0" newVersion="3.2.6.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="MySqlConnector" publicKeyToken="d33d3e53aa5f8c92" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-0.33.1.0" newVersion="0.33.1.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-10.0.0.0" newVersion="10.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

Apparently, at least in some cases, the problem appears to have something to do with dependent assemblies.

I ran into this issue as well today. I was able to workaround / fix my specific issue by removing all unnecessary assembly bindings in app.config. After doing this, my tests were discovered.

How did I determine my necessary bindings? I commented them all out and repeated these steps: (1) run my tests, and (2) reintroduced the binding hinted in the test failure. Luckily, I only had to repeat these steps for two bindings and not for all of my bindings (30+). /phew

Hope this helps.