Resolved

I can now confirm that the issue was caused by adding the “Dynatrace OneAgent” site extension to our function app in Azure. How exactly it breaks is unclear, but the process seems to do something that results in illegal awaits in the durable task framework. See more here: https://github.com/Azure/azure-functions-durable-extension/issues/2133#issuecomment-1343509726

Original post below

Description

My Durable Functions app has been running fine in Azure for while but has suddenly started to misbehave. Instances are completing their entire execution for the orchestrator function, however they are no longer completing their instance status. The statuses stay as “Running” and the “ExecutionCompleted” event is not added to the instance history table.

Expected behavior

At the end of an orchestration function, the instance should complete, fail or restart (ContinueAsNew).

Actual behavior

The orchestrator runs all the way through logically but the instance does not end correctly. I have found a single instance that has been marked as Completed since the issue appeared.

Relevant source code snippets

(Chopped up and simplified example to make sense of the screenshots)

        [FunctionName("Orchestrator")]
        public async Task RunOrchestrator(
            [OrchestrationTrigger] IDurableOrchestrationContext context,
            ILogger log)
        {
            var data = context.GetInput<OrchestratorInputModel>();
            try
            {
                if (data.Something)
                {
                    await context.CallActivityAsync("Calculate", data);
                    context.ContinueAsNew(data);
                    return;
                }
                await context.CallActivityAsync("Publish", data);
            }
            catch (SomeException exception)
            {
                await context.CallActivityAsync("UnrecoverableErrorHandler", exception);
            }
        }

Known workarounds

Redeploying to point to a new task hub worked for a few weeks before the problem returned.

App Details

• Microsoft.Azure.WebJobs.Extensions.DurableTask: 2.6.0
• Azure Functions runtime version: 3
• Programming language used: C#
• Framework: .NET Core 3.1

Screenshots

Columns

Correctly ending instance.

Instances that have run all the way through but are still running and missing the ExecutionCompleted line.

If deployed to Azure

• Timeframe issue observed: 2022-04-05T11:00:00Z onwards
• Azure storage account name: can share privately if requested
• Function App name: can share privately if requested
• Function name(s): Orchestrator
• Azure region: Australia East
• Orchestration instance ID(s):
  1. 00083b81c88b47db8e5829fe88858e3a
  2. 00f41cc833ce44d781e5b857a2fd8381
  3. 699e9886aea3416fbcdd92b11b8dbc84