SqlClient: Microsoft.Data.SqlClient.SNI.x64.dll crashes when using multiple AppDomains in IIS Express and IIS Server

I did some research and found the difference between my local environment and the environment on the pipeline that generates the SNI GA build. I found out that I was using an older version of the Windows 10 SDK 10.0.19041.0 and the pipeline vm has all the versions of the Window SDK installed including the latest i.e. 10.0.22000 and the SNI project references Windows 10 SDK without a specific version, so it uses the latest one installed. When I installed the latest Windows 10 SDK, I was able to reproduce the crash locally on my sample project and found out that the crash occurred inside of the registration of TraceLoggerProvider and compared the difference between the 2 versions of the Windows 10 SDK. It turns out a fatal error was introduced in the latest Windows 10 SDK version to prevent multiple registrations of the same TraceLoggingProvider (i.e. when using multiple AppDomain) due to a memory leak and memory corruption, so I’ve opened a PR to fix the multiple registrations issue. Hopefully, the fix will be included in the next 5.0 hotfix release and it might need to be back ported to the previous versions as well.

The PR has been merged and will be in the next release.

@JRahnama I was able to create a repro of this problem, though not involving IISExpress it sounds similar to the scenario described by @leonmeijer, which is also the problem we we’re seeing, and I suspect it has the same underlying cause.

Here is a small standalone repro that exhibits this problem when using AppDomains in .NET Framework 4.8:

Program.cs:

using System;
using Microsoft.Data.SqlClient;

namespace ConsoleApp1
{   
   public class Program
   {
      public const string ConnectionString = "Data Source=.;Initial Catalog=SinT.feature/sqlClient.ProtectionTest;Integrated Security=SSPI;Multiple Active Result Sets=True;Encrypt=False;Multi Subnet Failover=False";

      public class DbTest : MarshalByRefObject
      {
         public void Call()
         {
            using (SqlConnection conn = new SqlConnection(ConnectionString))
            {
               Console.WriteLine("Open connection");
               conn.Open();
            }
         }
      }

      static void Main(string[] args)
      {
         Console.WriteLine($"Start using {typeof(SqlConnection).Assembly.FullName}");
         AppDomain domain = AppDomain.CreateDomain("MyDomain");

         using (SqlConnection conn = new SqlConnection(ConnectionString))
         {
            Console.WriteLine("Open connection");
            conn.Open();

            DbTest dbTest = (DbTest)domain.CreateInstanceAndUnwrap(typeof(DbTest).Assembly.FullName, typeof(DbTest).FullName);
            try
            {
               dbTest.Call();
            }
            catch (Exception)
            {
               Console.WriteLine("Error");
            }

            Console.WriteLine("All calls done");
         }
      }
   }
}

ConsoleApp1.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net48</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
     <LangVersion>latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />
  </ItemGroup>

</Project>

And below is the output when running it, first with version 3.0.1 of Microsoft.Data.SqlClient, and then with version 4.1.0 (run on Windows 10 with latest updates in this scenario, but problem also observed on Windows Server 2019 and 2022)

PS> .\ConsoleApp1.exe
Start using Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5
Open connection
Open connection
All calls done

PS> $LASTEXITCODE
0

PS> .\ConsoleApp1.exe
Start using Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5
Open connection
Open connection

PS> $LASTEXITCODE
-1073740791

The event log contains the following error entry after the second execution:

Faulting application name: ConsoleApp1.exe, version: 1.0.0.0, time stamp: 0xdd7eed65
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x2bf4
Faulting application start time: 0x01d828badaa0788b
Faulting application path: F:\ETWork\SqlClientTest\ConsoleApp1\ConsoleApp1\bin\Debug\net48\ConsoleApp1.exe
Faulting module path: F:\ETWork\SqlClientTest\ConsoleApp1\ConsoleApp1\bin\Debug\net48\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 45bd024d-1012-4796-8dfc-3e0329cefe30
Faulting package full name: 
Faulting package-relative application ID: 

If running from Visual Studio with Native Debugging enabled and break on exceptions, we can see the following exception text at the call to connection.Open() from the other AppDomain:

Unhandled exception at 0x00007FFEBBBEC80C (Microsoft.Data.SqlClient.SNI.x64.dll) in ConsoleApp1.exe: An invalid parameter was passed to a function that considers invalid parameters fatal.

Two recent hotfixes 4.0.2 and 4.1.1 ship this fix. And, the next version 5.1 will be released till end of this year.

Release Notes

We are having the same issue as described by @alphaleonis. We are loading multiple appdomains in a Windows Service and executing EF queries and after we updated our dependencies our app is now crashing if 2 app domains are simultaniously making queries to the database.

Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c

With the code that can reproduce the issue provided by @alphaleonis is there an update yet on this issue? This should be regarded as a high priority bug IMHO.

Edit

I reverted my package version of Microsoft.Data.SqlClient.SNI back to 2.1.1 and the problem does not occur. I have not tried 3.0.0.

Sorry, I realized that I was testing a 5.0 dev SNI build, which worked. I just re-tried all 3 repro samples again using v5.0.0 from nuget, and it does continue to fail when multiple AppDomain instances are used. We’ll have to further investigate and get back to you soon.

@lcheunglci We’re still seeing the crash in v5.0.0 with the sample projects provided. With #1418 (comment) as an example, it’s still crashing on both x32 and x64 as shown below. The reproduction of the crash when using multiple appdomains is pretty consistent, but since most of the repros are console apps, you may not be seeing the crash unless you’re looking in the event log or output, since it blows up the whole process. This is an issue not just for console apps but also for any use in IIS which makes regular use of appdomains with a .NET Framework app.

Faulting application name: SqlClientCrashTest.exe, version: 1.0.0.0, time stamp: 0x932c7422
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 5.0.0.0, time stamp: 0x62e973e6
Exception code: 0xc0000409
Fault offset: 0x0000000000005b5d

The MDS v5.0.0 just went live, and I tried the 3 sample projects provided here: https://github.com/dotnet/SqlClient/issues/1418#issuecomment-1048797682 , https://github.com/dotnet/SqlClient/issues/1418#issuecomment-1121276135 , and https://github.com/dotnet/SqlClient/issues/1418#issuecomment-1121359393 and it no longer crashed in the Microsoft.Data.SqlClient.SNI.dll.

Not to belabor the issue, but I think that I too have been experiencing this issue… below is the abbreviated stacktrace I am seeing when debugging the static crash data in VS 2022. @lcheunglci - Could you (or someone) please verify that this is in fact the same issue being addressed in the coming 5.0 hotfix? This also may help others identify is this is in fact their issue as well. Any/All help is greatly appreciated. EDIT: adding @DavoudEshtehari

Microsoft.Data.SqlClient.SNI.x64.dll!SNITraceLogging::registerProvider(void)	
Microsoft.Data.SqlClient.SNI.x64.dll!SNIInitializeEx�()	
Microsoft.Data.SqlClient.SNI.x64.dll!SNIInitialize�()	

Yes, the fix will resolve this issue. As the issue lies in the registerProvider SNI method when registering the same provider more than once.

I did some research and found the difference between my local environment and the environment on the pipeline that generates the SNI GA build. I found out that I was using an older version of the Windows 10 SDK 10.0.19041.0 and the pipeline vm has all the versions of the Window SDK installed including the latest i.e. 10.0.22000 and the SNI project references Windows 10 SDK without a specific version, so it uses the latest one installed. When I installed the latest Windows 10 SDK, I was able to reproduce the crash locally on my sample project and found out that the crash occurred inside of the registration of TraceLoggerProvider and compared the difference between the 2 versions of the Windows 10 SDK. It turns out a fatal error was introduced in the latest Windows 10 SDK version to prevent multiple registrations of the same TraceLoggingProvider (i.e. when using multiple AppDomain) due to a memory leak and memory corruption, so I’ve opened a PR to fix the multiple registrations issue. Hopefully, the fix will be included in the next 5.0 hotfix release and it might need to be back ported to the previous versions as well.

Please backport to 4.0 as well, as we cannot uptake a non-LTS release.

My repro above exhibits the same issue with 5.0-preview3 as well.

Faulting application name: SqlClientCrashTest.exe, version: 1.0.0.0, time stamp: 0xfb0d886a
Faulting module name: Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.0.0.0, time stamp: 0x62a8f1f9
Exception code: 0xc0000409
Fault offset: 0x00015a32
Faulting process id: 0x525c
Faulting application start time: 0x01d8882523c1766d
Faulting application path: C:\Users\andre\Downloads\SqlClientCrashTest\SqlClientCrashTest\SqlClientCrashTest\bin\Debug\SqlClientCrashTest.exe
Faulting module path: C:\Users\andre\Downloads\SqlClientCrashTest\SqlClientCrashTest\SqlClientCrashTest\bin\Debug\Microsoft.Data.SqlClient.SNI.x86.dll
Report Id: 59beb78c-38b9-4802-ba5b-1d7e1b68e553
Faulting package full name: 
Faulting package-relative application ID: 

I think the MDS Team is currently in the middle of a release, and the samples provided should be sufficient to reproduce the issue since I was able to run into the same crash. From debugging the code, I believe I ran into a invalid parameter exception in the native side of MDS SNI.dll, but wasn’t able to dive much deeper since I would need the pdb files to the SNI.dll included, so I wasn’t able to extract much info out of it. I don’t think the SNI code is open sourced, so only the MDS team is able to further investigate, so it’s currently on hold on again.

Thanks @HamsterExAstris for providing the repro sample. I was able able to reproduce what you mentioned with it and I’ll debug and diff the changes between v3.1 and v4.1 and see if I could figure out what changes caused it to break in 4.1.

I’m glad to announce that Microsoft.Data.SqlClient v5.0.1 has been released.

A patch for 5.0 is needed asap as well.

From my experience with an access violation error, it could be a mismatch between the version of MDS.SNI and MDS.

The thing is that I get this crash with MDS 4.1.0 w/ MDS.SNI 4.0.0, but it seems to work when using MDS 4.1.0 w/ MDS.SNI 3.0.0 😅

This issue is still tagged as waiting for customer and needs more info. It looks like repro samples have been provided. Does the MDS team still need more information to fix this?

Here is a repo with ASP.NET MVC basic project template reproducing the crash of IIS (https://github.com/antymon4o/SqlClient_issue_1418). The Connection is opened in ValuesController, action Get(). You should set (hardcode) the connection string in ValuesController.cs file. The configuration of the IIS site is detailed in the README file of the repo. One application pool is used for the site and virtual application resulting in loading Microsoft.Data.SqlClient.SNI.x64.dll in one process but in different application domains. After the crash in Event Viewer there is this Error

Faulting application name: w3wp.exe, version: 10.0.17763.1, time stamp: 0xcfdb13d8
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x1dbc
Faulting application start time: 0x01d863c2cf5ce98c
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: L:\WebApplication1\bin\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 88a6c161-9ff2-46c6-b6e4-d70388fca3ef
Faulting package full name: 
Faulting package-relative application ID: 

Here’s a console-based repro. (Sorry it’s not quite as minimal as the ones above.) No MARS or MSF in use. It works with MDS v2.1.1 and v3.1, and in v4.1 with only one AppDomain, but bombs on v4.1 with two AppDomains.

SqlClientCrashTest.zip

@lcheunglci Thank you for investigating this.

FWIW, we aren’t using MARS or MSF with MDS and we’re experiencing this crash with MDS 4.1. We’re currently running MDS 3.1 in production.

We recently upgraded our ASP.NET / .NET Framework 4.8 web app to use Microsoft.Data.SqlClient Version 4.1 and Microsoft.Data.SqlClient.SNI Version 4.0.

The new package worked fine locally and in our development environment, but when we deployed our app to our QA server the IIS application pool immediately crashes when starting up:

Faulting application name: w3wp.exe, version: 10.0.17763.1, time stamp: 0xcfdb13d8
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x292c
Faulting application start time: 0x01d84dbf4eb092c5
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: c:\inetpub\wwwroot\qa-edelweissplus\bin\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 2ce9eb27-a486-4986-abaf-94fcf9eeac37
Faulting package full name: 
Faulting package-relative application ID: 

We upgraded primarily because we lost SQL statement logging using Application Insights with Microsoft.Data.SqlClient 3.0 and upgrading to 4.0 seemed to fix it.

@JRahnama [Serializable] doesn’t make a difference if the DbTest class derives from MarshalByRefObject. Of course, this sample works if we add [Serializable] and remove the inheritance from MarshalByRefObject, but then we’re testing something else entirely. (The problem occurrs when we have SqlConnections opened in multiple AppDomains)