azure-functions-dotnet-worker: Graceful shutdown of a QueueTrigger function via observing the CancellationToken

Description

I ran into an issue with a long running Azure Function QueueTrigger that will start executing, but somewhere in the middle of execution, we’ll see a log message that the application is shutting down, and there are no errors or exceptions logged. The application is then left in a weird state.

2024-06-05 02:31:15.782 [Information] [redacted legitimate activity, 1/250]
2024-06-05 02:31:15.860 [Information] [redacted legitimate activity, 2/250]
2024-06-05 02:31:15.866 [Information] [redacted legitimate activity, 3/250]
2024-06-05 02:31:16.233 [Information] [Microsoft.Hosting.Lifetime] Application is shutting down...
2024-06-05 02:31:31.133 [Information] [Microsoft.Hosting.Lifetime] Application started. Press Ctrl+C to shut down.

Inside my function, the CancellationToken.IsCancellationRequested property is never true, so I can’t tell when the application is shutting down.

When running locally via Visual Studio, I tried to simulate shutting down the function by hitting CTRL+C, but the code that checks for cancellation is never hit.

When running a similar function in Azure, I tried to simulate shutting down by disabling the function in Azure Portal while the function was executing a long-running operation. The code that checks for and logs cancellation is never hit.

Questions:

  • Why can’t I observe a cancellation request inside my QueueTrigger function?
  • Is there some other way to be notified that the function runtime is shutting down?
[Function(nameof(Function1))]
public async Task Run([QueueTrigger("test-queue")] QueueMessage message, CancellationToken cancellationToken)
{
    _logger.LogInformation($"C# Queue trigger function processed: {message.MessageText}");
    _logger.LogInformation("cancellationToken == CancellationToken.None? {IsEqual}", cancellationToken == CancellationToken.None);

    cancellationToken.Register(() =>
    {
        Console.WriteLine("CancellationToken Register callback invoked.");
    });

    foreach (var ix in Enumerable.Range(start: 0, count: 1_000))
    {
        await Task.Delay(1_000);

        if (cancellationToken.IsCancellationRequested)
        {
            _logger.LogInformation("Cancellation requested!");
            return;
        }
        else
        {
            _logger.LogInformation("Cancellation not requested.");
        }
    }
}

Related

Technical Details

This is a v4 Azure Function:

  • Visual Studio 2022 17.11.0 Preview 1.1
  • Runtime: dotnet-isolated
  • .NET: 8.0.6
  • Function type: QueueTrigger
  • Consumption Plan

Steps to reproduce

I have a full reproduction in this GitHub repo:

  1. Open the solution in Visual Studio
  2. Add a local.settings.json file to the project root as described in the README
  3. Run the application
  4. Add a simple queue message to trigger the function
  5. Hit CTRL+C to stop the function from executing

Expected: the CancellationToken.IsCancellationRequested property eventually becomes true, and I can use that to gracefully exit my application.

Actual: the CancellationToken.IsCancellationRequested property is always false.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 18 (2 by maintainers)

Most upvoted comments

This is NOT a local development issue. We are using .NET 8 Queue Triggered Azure Functions with Flex Consumption Plan as mentioned above, the function host always requests a cancellation somewhere in the middle of the execution. However, we have similar Queue Triggered .NET 8 functions with the basic Consumption Plan which are working properly.

@kshyju It’s not a local development issue. see my comment based on the azure function with flex consumption. this is tested within an azure deployment not locally. maybe it’s the same reason but it should be very easy to reproduce.

There seems to be a gap in the local development experience for out-of-process language workers in this use case (simulating host stop via core tools). I’ve opened a new GitHub issue to track the work on this. Please follow the issue here:

Gracefully stop the functions host and propagate cancellation tokens

Closing this thread as further updates will be reflected in that issue.

Thanks for your patience, everyone. I’ll investigate and follow up soon.

@jonsagara Thanks for sharing the repro app.

Hi! I have a similar issue using Flex Consumption Plan with Queue Trigger. In the middle of the exectuion after 30-40 secs the function host requests a cancellation so my function shuts down. To solve this I had to pass CancellationToken.None to the downstream calls instead of the real CancellationToken coming from the function entry. @ToFi79 are you saying this is because of the scaling? We had only one instance running at the time.

I just updated my test project and confirmed that the same behavior still exists in .NET 9 with the latest nuget packages:

https://github.com/jonsagara/AzureFunctionQueueTriggerCancellationTokenTest

TLDR: The CancellationToken works as expected (allows me to exit the foreach loop) in the .NET 6 in-process version of the Azure Function. It does not work in the .NET 8 or .NET 6 isolated versions.

I updated my reproduction app to have three different projects:

  • .NET 8 Isolated
  • .NET 6 Isolated
  • .NET 6 In Process

I ran each of them and tried to cancel execution while inside the foreach loop.

.NET 8 Isolated Results

When running the QueueTriggerCancellationTokenTestNET8Isolated function, I get the following output:

[2024-06-13T21:32:36.207Z] Host lock lease acquired by instance ID '0000000000000000000000005864DAB7'.
[2024-06-13T21:32:39.153Z] Executing 'Functions.Function1' (Reason='New queue message detected on 'test-queue'.', Id=a33717ef-5fa9-4d3a-8c25-bc04502b9722)
[2024-06-13T21:32:39.156Z] Trigger Details: MessageId: c438524d-2a36-4b56-9aa5-ca9f56f0ed9e, DequeueCount: 1, InsertedOn: 2024-06-13T21:32:39.000+00:00
[2024-06-13T21:32:39.243Z] context.CancellationToken == cancellationToken? True
[2024-06-13T21:32:39.243Z] context.CancellationToken == CancellationToken.None? False
[2024-06-13T21:32:39.243Z] hostAppLifetime is null
[2024-06-13T21:32:39.243Z] C# Queue trigger function processed: { "name": "jon" }
[2024-06-13T21:32:39.243Z] cancellationToken == CancellationToken.None? False
[2024-06-13T21:32:40.242Z] Cancellation not requested.
[2024-06-13T21:32:41.245Z] Cancellation not requested.
[2024-06-13T21:32:42.251Z] Cancellation not requested.
[2024-06-13T21:32:43.255Z] Cancellation not requested.
[2024-06-13T21:32:44.259Z] Cancellation not requested.
[2024-06-13T21:32:45.264Z] Cancellation not requested.
[2024-06-13T21:32:46.270Z] Cancellation not requested.
[2024-06-13T21:32:47.276Z] Cancellation not requested.
[2024-06-13T21:32:48.282Z] Cancellation not requested.
[2024-06-13T21:32:49.290Z] Cancellation not requested.
[2024-06-13T21:32:50.294Z] Cancellation not requested.
[2024-06-13T21:32:50.828Z] Host did not shutdown within its allotted time.

cancellationToken.IsCancellationRequested is never true.

.NET 6 Isolated Results

When running QueueTriggerCancellationTokenTestNET6Isolated, I get the following output:

[2024-06-13T21:35:35.192Z] Host lock lease acquired by instance ID '0000000000000000000000005864DAB7'.
[2024-06-13T21:35:36.088Z] Executing 'Functions.Function1' (Reason='New queue message detected on 'test-queue'.', Id=591cb885-3d07-42f0-9cda-0167b5018829)
[2024-06-13T21:35:36.095Z] Trigger Details: MessageId: 21470f39-e98b-49d9-a509-96761ec3ed2e, DequeueCount: 1, InsertedOn: 2024-06-13T21:35:36.000+00:00
[2024-06-13T21:35:36.204Z] context.CancellationToken == CancellationToken.None? False
[2024-06-13T21:35:36.204Z] hostAppLifetime is null
[2024-06-13T21:35:36.204Z] cancellationToken == CancellationToken.None? False
[2024-06-13T21:35:36.204Z] C# Queue trigger function processed: { "name": "jon" }
[2024-06-13T21:35:36.204Z] context.CancellationToken == cancellationToken? True
[2024-06-13T21:35:37.207Z] Cancellation not requested.
[2024-06-13T21:35:38.212Z] Cancellation not requested.
[2024-06-13T21:35:39.233Z] Cancellation not requested.
[2024-06-13T21:35:40.243Z] Cancellation not requested.
[2024-06-13T21:35:41.250Z] Cancellation not requested.
[2024-06-13T21:35:42.255Z] Cancellation not requested.
[2024-06-13T21:35:43.259Z] Cancellation not requested.
[2024-06-13T21:35:44.279Z] Cancellation not requested.
[2024-06-13T21:35:45.284Z] Cancellation not requested.
[2024-06-13T21:35:46.289Z] Cancellation not requested.
[2024-06-13T21:35:47.293Z] Cancellation not requested.
[2024-06-13T21:35:47.654Z] Host did not shutdown within its allotted time.
[2024-06-13T21:35:47.663Z] FunctionContext CancellationToken Register callback invoked.
[2024-06-13T21:35:47.669Z] CancellationToken Register callback invoked.

cancellationToken.IsCancellationRequested is never true inside the loop, but you can see that the CancellationToken.Register callback is eventually invoked just before the function completes.

.NET 6 In Process Results

When running QueueTriggerCancellationTokenTestNET6InProcess, I get the following output:

[2024-06-13T21:37:21.067Z] Host lock lease acquired by instance ID '0000000000000000000000005864DAB7'.
[2024-06-13T21:37:22.696Z] Executing 'Function1' (Reason='New queue message detected on 'test-queue'.', Id=42cf89d3-c1a1-49ff-a75a-875821b11d30)
[2024-06-13T21:37:22.698Z] Trigger Details: MessageId: b5c4ed8b-413e-4345-953e-d63d96b3a5b0, DequeueCount: 1, InsertedOn: 2024-06-13T21:37:21.000+00:00
[2024-06-13T21:37:22.709Z] C# Queue trigger function processed: { "name": "jon" }
[2024-06-13T21:37:22.710Z] cancellationToken == CancellationToken.None? False
[2024-06-13T21:37:23.719Z] Cancellation not requested.
CancellationToken Register callback invoked.
[2024-06-13T21:37:24.723Z] Cancellation requested!
[2024-06-13T21:37:24.753Z] Executed 'Function1' (Succeeded, Id=42cf89d3-c1a1-49ff-a75a-875821b11d30, Duration=2078ms)

It works in the .NET 6 in-process model.

Immediately after I press CTRL+C, you can see that cancellationToken.IsCancellationRequested is true, and the loop exits.