aspnetcore: .NET Runtime Exception: ObjectDisposedException

Describe the bug

After upgrading to ASP.NET Core 2.2, we frequently face this error and our website gets restarted. This happens once an hour or so. We had to target 2.1 again to fix this issue. Here are the errors from event viewer (in the order they have occurred):

Application: dotnet.exe
CoreCLR Version: 4.6.27110.4
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at System.IO.FileStream.FileStreamCompletionSource.CompleteCallback(UInt64 packedResult)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Faulting application name: dotnet.exe, version: 2.2.27110.6, time stamp: 0x5be769ed
Faulting module name: KERNELBASE.dll, version: 10.0.14393.2457, time stamp: 0x5b7e2adb
Exception code: 0xe0434352
Fault offset: 0x0000000000033c58
Faulting process id: 0x34374
Faulting application start time: 0x01d48cab8dc6d688
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Faulting package full name: 
Faulting package-relative application ID:

Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: C:\Program Files\dotnet\dotnet.exe
P2: 2.2.27110.6
P3: 5be769ed
P4: System.Private.CoreLib
P5: 4.6.27110.4
P6: 5be75665
P7: 27fd
P8: b
P9: System.ObjectDisposedException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_C__Program Files_55e13f92ac2eada5e8dd2daa7dea78fd1e8d5d_0e1906bf_ade641d7

Analysis symbol: 
Rechecking for solution: 0
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Report Status: 4
Hashed bucket:

Fault bucket 1396378374383491433, type 5
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: C:\Program Files\dotnet\dotnet.exe
P2: 2.2.27110.6
P3: 5be769ed
P4: System.Private.CoreLib
P5: 4.6.27110.4
P6: 5be75665
P7: 27fd
P8: b
P9: System.ObjectDisposedException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_C__Program Files_55e13f92ac2eada5e8dd2daa7dea78fd1e8d5d_0e1906bf_69a648dc

Analysis symbol: 
Rechecking for solution: 0
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Report Status: 0
Hashed bucket: 0fdfa957dcca13d25360eebbb12a5969

To Reproduce

Steps to reproduce the behavior: Use SDK v2.2.100 with IIS on Windows Server 2016

Additional context

Add any other context about the problem here. Include the output of dotnet --info

Host (useful for support):
  Version: 2.2.0
  Commit:  1249f08fed

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 55 (27 by maintainers)

Most upvoted comments

@jrmoreno1 I can confirm that I had the same issue and this is resolved in 3.0.

I would still like to point out that there’s so many breaking changes in 3.0 that it’s not really an easy option for everyone to migrate just to resolve a pressing production issue.

gaborbuzasi on Oct 16, 2019

I see that this issue was patched for 3.0. Will there be a patch 2.x line?

rgelb on Sep 6, 2019

Right, preview5 is the next regular 3.0 release. We’re also considering this for a patch in 2.2.6 or later.

Tratcher on Apr 17, 2019

I’m running on Azure AppServices. I’m getting this issue when running InProc.

Here’s output from Kudo (although I tried to run 2.2.103 with self contained, no difference) :

Kudu Remote Execution Console
Type 'exit' then hit 'enter' to get a new powershell process.
Type 'cls' to clear the console

PS D:\home> dotnet --version
dotnet --version
2.2.102

PS D:\home> dotnet --info
dotnet --info
.NET Core SDK (reflecting any global.json):

 Version:   2.2.102

 Commit:    96ff75a873



Runtime Environment:

 OS Name:     Windows

 OS Version:  10.0.14393

 OS Platform: Windows

 RID:         win10-x86

 Base Path:   D:\Program Files (x86)\dotnet\sdk\2.2.102\



Host (useful for support):

  Version: 2.2.1

  Commit:  878dd11e62



.NET Core SDKs installed:

  1.1.10 [D:\Program Files (x86)\dotnet\sdk]

  2.1.500 [D:\Program Files (x86)\dotnet\sdk]

  2.1.503 [D:\Program Files (x86)\dotnet\sdk]

  2.2.102 [D:\Program Files (x86)\dotnet\sdk]



.NET Core runtimes installed:

  Microsoft.AspNetCore.All 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

  Microsoft.AspNetCore.All 2.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

  Microsoft.AspNetCore.All 2.2.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

  Microsoft.AspNetCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

  Microsoft.AspNetCore.App 2.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

  Microsoft.AspNetCore.App 2.2.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

  Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.2.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]



To install additional .NET Core runtimes or SDKs:

  https://aka.ms/dotnet-download

PS D:\home>

My exception is as follows:

System.ObjectDisposedException at System.Threading.CancellationTokenSource.ThrowObjectDisposedException
System.Threading.CancellationTokenSource.ThrowObjectDisposedException line:0
Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<>c__DisplayClass314_0.<AbortIO>b__0 line:0

abergs on Feb 12, 2019

I’ve seen the following exception 80 times in the last 7 days on a site that gets about 20K page views per week. Windows 2012 R2, dotnet hosting 2.2.6, InProcess.

SourceContext
Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer

Exception
System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at System.Threading.CancellationTokenSource.Cancel()
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.<>c__DisplayClass314_0.<AbortIO>b__0(Object t)

jeremycook on Jul 22, 2019

@jrmoreno1 can you try running 3.0? This issue should be resolved there.

jkotalik on Oct 16, 2019

The problem is that you may have solved that the process doesn’t crash, but when this error happens the response from the API isn’t returned and it clears out all the CORS headers too. So users of our API need to keep refreshing their application until the error doesn’t come up. I don’t think upgrading to a preview version of a framework (3.0) is a solution to this.

Even before the fix in 3.0, the underlying connection must have already been aborted for the CancellationTokenSource backing HttpContext.RequestAborted to become disposed leading to an ObjectDisposedException. This means that clearing out the CORS headers should be a non-issue since the client could have never received said headers over a closed connection in the first place.

If you are seeing a bunch of aborted requests and users need to keep refreshing the application to get a response, that points to a deeper issue. The ObjectDisposedException is merely a symptom of the already-aborted requests.

I would look at info-level and higher ASP.NET core logs to see if there’s any indication why the connections associated with the failing requests are being closed.

halter73 on Sep 13, 2019

I would be very curious if there are any updates on this one related to .NET Core 2.2.x users? @Tratcher @shirhatti

gaborbuzasi on Sep 10, 2019

@Tratcher I couldn’t see this issue mentioned in the 2.2.6 release notes. Did I miss it or will it not be released for 2.x?

abergs on Jul 22, 2019

@vytautask The bug you’re observing (which is that the CancellationTokenSource backing the RequestAborted token can be disposed prior to middleware completing) still exists.

@Tratcher is referring specifically to a fix for referencing CancellationTokenRegistration.Token after the CancellationTokenSource has been disposed. This prevents the process crash and made it in to 2.2.1. We will need to rework #4447 for the full fix.

halter73 on Mar 13, 2019

Do you have any idea what application code or event triggered this?

This may need to be moved to https://github.com/dotnet/corefx/issues.

Tratcher on Dec 5, 2018