aspnetcore: SIGTERM signal in .NET 6 app deployed in azure Kubernetes service is not triggered

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

I have a .NET 6 application. This application contains a BackgroundService that is used to handler background jobs. These jobs are fired on a certain time based on different schedules.

I need to implement graceful shutdown when the application is stopping, because there might be some running jobs and I want to complete the jobs before stopping the application.

The app is deployed in Azure Kubernetes Service, when running this command kubectl delete pod {podname} to delete a certain pod, I am not able to handle the sigterm and write my logic to handle graceful shutdown.

I am using the IHostApplicationLifetime and registering on the ApplicationStopping event.

IHostApplicationLifetime _applicationLifetime; _applicationLifetime.ApplicationStopping.Register(OnAppStopping);

In the docker file I added the below line

STOPSIGNAL SIGTERM

The OnAppStopping is never fired on AKS.

Expected Behavior

The ApplicationStopping event should be triggered in order to handle graceful stop of the application

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

Anything else?

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 26 (18 by maintainers)

Most upvoted comments

I used a minimal k8s config - please let me know if you’re doing something different.

apiVersion: v1
kind: Pod
metadata:
  name: gh45801pod
spec:
  containers:
  - name: gh45801container
    image: ourserver.azurecr.io/gh45801test:latest

amcasey on Jan 20, 2023

And from AKS

2023-01-20T23:12:27.931498547Z In the StartAsync of TestBackgroundService
2023-01-20T23:12:27.936843273Z [40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]
2023-01-20T23:12:27.936857574Z       Application started. Press Ctrl+C to shut down.
2023-01-20T23:12:27.937568377Z [40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]
2023-01-20T23:12:27.937578877Z       Hosting environment: Production
2023-01-20T23:12:27.937685178Z [40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]
2023-01-20T23:12:27.937692778Z       Content root path: /app
2023-01-20T23:13:12.235031704Z In the OnApplicationStopping
2023-01-20T23:13:12.235188805Z [40m[32minfo[39m[22m[49m: Microsoft.Hosting.Lifetime[0]
2023-01-20T23:13:12.235218305Z       Application is shutting down...
2023-01-20T23:13:12.242233053Z In the StopAsync of TestBackgroundService
2023-01-20T23:13:12.242326854Z In the OnApplicationStopped

(The color markers appear to have been cleaned up in .net core 7)

amcasey on Jan 20, 2023

Yes, that is right, when using docker support from visual studio, It deploys to docker and attach and debugger. The ApplicationStarted event is firing normally, however the ApplicationStopping is not.

Due to the way VS runs docker containers, your app will never get SIGTERM if the container is run from VS (even if you’re not debugging). I understand this is part of optimizing the inner dev loop to not require restarting the container every launch.

However, if you run the container another way, e.g. by right clicking on the dockerfile and selecting “Build docker image” and then running the resulting image (e.g. in Docker Desktop), you should see logger and/or console output when you send SIGTERM to the container.

Sorry this is such a gotcha - I spent a while scratching my head over it myself.

amcasey on Jan 20, 2023

We’re all on vacation. Expect updates in a couple of weeks.

davidfowl on Dec 20, 2022

same here, any update on this?

karimMogh on Dec 16, 2022

Note that I reproduced the same scenario on docker Linux. I sent a SIGTERM signal to the container using the below cammand. docker kill --signal=“SIGTERM” container-id/name

This is the sort of information we need to reproduce the issue (see the “steps to reproduce” part of the issue above).

Find the attached k8s manifest k8s.txt

This isn’t a k8s manifest, but it seems like you can reproduce the issue on with docker kill so that will be the repro.

I have 2 more questions:

https://github.com/Ghyath-Serhal/SigTermKuber/blob/main/SigTermKubernetes/Dockerfile#L5-L6 - Why does your docker file expose ports even though this application isn’t exposing any?
How are you observing with docker that OnApplicationStopping isn’t firing? Are you using docker logs?

davidfowl on Dec 5, 2022