aspnetcore: Failed to gracefully shutdown application (IIS Hang)

on application pool recycle (automatic by duration) and under high load i often get the following entry in event log Failed to gracefully shutdown application 'MACHINE/WEBROOT/APPHOST/WEBNAME'.

this was already happen in 2.1 with outproc, but now i’ve updated to 2.2 with hostingModel=“InProcess”, but i get a additional eventlog entry:

CoreCLR Version: 4.6.27207.3
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.IOException: The handle is invalid
   at System.ConsolePal.WindowsConsoleStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.WriteLine(String value)
   at System.IO.SyncTextWriter.WriteLine(String value)
   at System.Console.WriteLine(String value)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.<>c__DisplayClass6_0.<AttachCtrlcSigtermShutdown>g__Shutdown|0()
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.<>c__DisplayClass6_0.<AttachCtrlcSigtermShutdown>b__1(Object sender, EventArgs eventArgs)
   at System.EventHandler.Invoke(Object sender, EventArgs e)
   at System.AppContext.OnProcessExit(Object sender, EventArgs e)

I use overlapped recycling, the new process is already stated, but i need to recycle manualy else the IIS is not processing requests anymore, they just timed out.

Setup:

  • Windows Server 2012 R2
  • IIS 8.5
  • .net core runtime and hosting bundle 2.2.1
  • overlapped recycling enabled
  • web: preload enabled = true
  • application pool: start mode always running
  • 32bit application

any idea why this exception happens? the related code should be this https://github.com/aspnet/Hosting/blob/master/src/Microsoft.AspNetCore.Hosting/WebHostExtensions.cs#L142 (this Console.Log)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (10 by maintainers)

Most upvoted comments

To expand on what @deanward81 said: What are the chances of this getting fixed in 2.2? 3.0 is great, and we’re trialing the preview in some areas, but this is crashing the process on the latest stable version.

This is affecting every application we’re porting at Stack Overflow. While we’ll upgrade to 3.0 as soon as is reasonable, troubleshooting, logging, monitoring, or anything to do with process crashes and event logs is a very high noise to signal ratio and triggers every build.

Can we please get a fix for this in 2.2? By fix I mean something where the process doesn’t crash every shutdown, it doesn’t have to be fancy. Right now we can’t alert on process crashes because every deploy would have n of them, and that makes the platform a risky proposition to move to. The impact of this issue has become more of an issue the bigger the service and when debugging anything, so we now consider this a blocker to any further production deploys. Even a hacky workaround on current bits would be very much appreciated - anything to unblock our moves.

this is happening to us as well. we are load balanced, and as soon as one instance fails from this, the others shortly fail as well. This issue is happening during high traffic hours