azure-functions-durable-extension: Durable functions created hanging in pending state for long time, start up days later.

Description

Our durable functions are getting stuck in the pending state, sitting like that for days, and then magically starting. This started after upgrading the nuget package versions, but I haven’t been able to fix the issue by redeploying the old code, so I’m not sure if it’s related.

Here are the package changes that were made: Microsoft.Azure.DurableTask.Core 2.1.0 => 2.1.2 Microsoft.Azure.Management.Fluent 1.21.0 => 1.24.0 Microsoft.Azure.Management.ResourceManager.Fluent 1.21.0 => 1.24.0 Microsoft.Azure.Services.AppAuthentication 1.2.0-preview2 => 1.3.0 Microsoft.Azure.WebJobs.Extensions.DurableTask 1.8.0 => 1.8.3 Microsoft.EntityFrameworkCore.Design 2.2.4 => 2.2.6 Microsoft.EntityFrameworkCore.SqlServer 2.2.4 => 2.2.6 Microsoft.Graph 1.15.0 => 1.16.0 Microsoft.IdentityModel.Protocols.OpenIdConnect 5.4.0 => 5.5.0 Microsoft.IdentityModel.Tokens 5.4.0 => 5.5.0 Microsoft.NET.Sdk.Functions 1.0.24 => 1.0.29

The application consists of a UI front end that is calling an HTTP triggered function that starts a durable function.

I also have the same version of this code running in a separate function app that is not (currently) experiencing this issue.

Expected behavior

Durable functions would start in a timely fashion.

Actual behavior

Durable functions are sitting in the pending state for days before running

Relevant source code snippets

string instanceId = await orchestrationClient.StartNewAsync("SelfServiceRequestOrchestrator", ssRequest);

Known workarounds

None

App Details

• Durable Functions extension version (e.g. v1.8.3): 1.8.3
• Azure Functions runtime version (1.0 or 2.0): 2.0
• Programming language used: csharp

Screenshots

If deployed to Azure

We have access to a lot of telemetry that can help with investigations. Please provide as much of the following information as you can to help us investigate!

• Timeframe issue observed: (see below)
• Function App name:
• Function name(s): SelfServiceRequestOrchestrator
• Azure region: East US 2
• Orchestration instance ID(s): (see below)
• Azure storage account name:

Request 1

• Status: Started successfully
• InstanceID: a4b34ae1d8be40309db77ca57e77b04f
• Time: 2019-08-14 17:28:04 (UTC)

Request 2

• Status: hung at pending state
• InstanceID: ef433fea4f074da49b00ce333725547c
• Time: 2019-08-14 17:29:14 (UTC)

Request 3

• Status: Started successfully
• InstanceID: ad0f0a15e54648db9fb16faa63ef4999
• Time: 2019-08-14 17:29:40 (UTC)