runtime: AppDomain.ProcessExit is not invoked on docker stop
Describe the bug
Running a Generic Host that uses ConsoleLifetime in a Windows Container (nanoserver/.NET Core or servercore/.NET Framework) doesn’t do graceful shutdown when the container is stopped - it just terminates the process.
To Reproduce
Steps to reproduce the behavior: I’m using VS 2019 with .NET Core 2.2(.6) though I believe this repros with .NET Core 3.0.0-preview6 as well.
Program.cs:
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System;
class Program
{
static void Main()
{
using (IHost host = new HostBuilder().ConfigureLogging(l => l.AddConsole()).Build())
{
host.Start();
host.WaitForShutdown();
Console.WriteLine("Shutting down gracefully.");
}
}
}
App.csproj:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.2</TargetFramework>
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
<SelfContained>false</SelfContained>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Hosting" Version="2.2.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.2.0" />
</ItemGroup>
</Project>
docker run -v X:\path\to\source\bin\Release\netcoreapp2.2\win-x64\publish\:C:\app mcr.microsoft.com/dotnet/core/runtime:2.2.6-nanoserver-1809 C:\app\App.exe
(wait for container to print started message)
docker container stop <id/alias>
Expected behavior
Container output shows “Shutting down gracefully.”
Actual behavior
Container terminates without showing that message.
Additional context
It looks like the AppDomain.ProcessExit event doesn’t trigger during any Windows container shutdown (regardless of nanoserver vs servercore and .NET Core vs .NET Framework of any version), though I didn’t test all combinations.
With 1809, I can get a graceful shutdown notification from the OS directly by P/Invoking SetConsoleCtrlHandler and looking for a callback for CTRL_SHUTDOWN_EVENT.
Note that this event does not get delivered if docker is run with the -t (tty) flag (that interferes with delivering the shutdown signal - see the moby issue for details).
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 2
- Comments: 32 (18 by maintainers)
Links to this issue
Commits related to this issue
- Handle windows logoff and shutdown events By doing a clean EEShutdown, including calling ProcessExit handlers. This allows ProcessExit to respond to 'docker stop', which sends CTRLSHUTDOWN_EVENT to w... — committed to sbomer/runtime by sbomer 4 years ago
- Handle windows shutdown event (#41101) * Handle windows logoff and shutdown events By doing a clean EEShutdown, including calling ProcessExit handlers. This allows ProcessExit to respond to 'dock... — committed to dotnet/runtime by sbomer 4 years ago
- Handle windows logoff and shutdown events By doing a clean EEShutdown, including calling ProcessExit handlers. This allows ProcessExit to respond to 'docker stop', which sends CTRLSHUTDOWN_EVENT to w... — committed to dotnet/runtime by sbomer 4 years ago
- [release/5.0] Handle windows shutdown event (#41182) * Handle windows logoff and shutdown events By doing a clean EEShutdown, including calling ProcessExit handlers. This allows ProcessExit to re... — committed to dotnet/runtime by github-actions[bot] 4 years ago
- Runtime ingestion (#73) * Applied BinaryPrimitives throughout CoreLib (#37582) * Flaky multiple HTTP/2 connection tests read full request body (#40844) There are 2 flaky HTTP/2 tests verifying ... — committed to dotnet/runtimelab by joperezr 4 years ago
The issue that running .NET Core on Docker does not handle shutdown, makes it very advisable to avoid using .NET Core in relation with Docker.
I have searched for hours and looked through countless issues, not able to find any good solution to this problem. Have not found any way of doing a graceful shutdown of .NET Core 3.0 apps running on Docker.
The issue popped up on Linux when the code very often made a database corrupt due to non-graceful shutdown. After this I have done a lot of debugging and research with Docker Desktop on Windows to find a solution, but none so far.
Creating the most basic “Worker Service” project in Visual Studio 2019 and .NET Core 3.0, checking the “Docker” support, and then running on Docker - then running docker stop, should trigger at least some event ASAP in the running code - but there is none, have not found any way to listen, it just eventually is killed and dies horribly.
This basically stops anyone from doing any sensible and production grade deployment of .NET Core 3.0 on Docker. Unless your application is 100% stateless and involves no other local resources, it can be very problematic.
I hope someone comes up with a solution that is easy to implement, at the current state it is not good at all.
This has been an ongoing issue since 2016 from all the github issues I read all over, and looking at the issue referenced above here as well, it clearly says:
That was posted in September 2017. That is more than 2 years ago.
Though that issue is related to Windows Containers, but the same issue applies to Linux containers.
Anyone else found a usable solution to the issue?
What is the guidance from Microsoft? Should we avoid using Docker until a solution is found? When will that happen, as this has been ongoing for many years in various forms already.
@agocke @vitek-karas is this on your radar for 5.0? This and linked issues seem to have interest.
@sbomer Could you take a look at this?
It was on my radar, but my understanding is that it only affects a small number of Windows container issues. Since this seems to be bigger than I thought I’ll see if we can get to this issue soon.
We would really like to see this in .NET Core 3.1 as we are driving a large fleet of our services from .NET Framework hosted in Cloud Services to .NET Core.
There are a few crossed-paths going on here. This thread is specific to Windows containers, so let’s keep it focused on that. I’ll move this over to runtime to track future improvements (likely via hooking CTRL_SHUTDOWN_EVENT as suggested above). If there are issues with Hosting on Linux containers, I’d suggest filing a separate issue on dotnet/runtime for that.
FYI, This is my workaround to send signal when docker debugging by Visual Studio.
click ‘open terminal window’ in the container view to enter into the docker container.
In the terminal, you enter
pidof dotnetto get pid of entry process.then you enter
kill -s SIGTERM <PID>with pid.Note: Im using docker with ubuntu based image, and entrypoint like
ENTRYPOINT ["dotnet", "myapp.dll"].Have done further investigation of my initial issue with ungraceful shutdowns. The issue was that shutdown was signaled, and the container instance started logging, but shutdown took more than 10 seconds, so it was killed before it was able to flush any log messages indicating that shutdown was being signaled.
So after debugging (tailing) the logs, and then comparing to logs on disk, it is clear that the container is never able to flush when shutdown only has 10 seconds to process.
Using the -t parameter to increase shutdown, made it work as expected.
(This is on Ubuntu 19.10, not Windows)
So primarily the main issue here is that debugging from Visual Studio, no signals are sent from the Docker runtime to your running container. I hope that will be added soon, as it can create issues with resource disposal during development and debugging.