azure-functions-durable-extension: Orchestration instance started stuck in pending status (Functions v2 breaking change)
I have a timer that fires ever 6 hours. It triggers a orchestration starter function that checks for an existing instance and starts/terminates/restarts the orchestration.
Recently the trigger log output for each trigger event has indicated an existing instance was found in pending status and with a last update time equal to its start time (6 hours previously from the last trigger event).
Here is a representative trigger function output.
Triggered by <Timer>
Getting orchestrator state from table storage...
Found LastKnownInstanceID: b9ec9716-2768-4775-a84b-522440fa6240
Existing instance found with name Orchestrator
Started at 6/12/2018 12:00:01 PM
Last updated at 6/12/2018 12:00:01 PM
Current status: Pending
Termination requested for reason: <Existing instance passed time limit>
Terminating existing instance with instanceID <b9ec9716-2768-4775-a84b-522440fa6240>...
Instance terminated
Starting new instance of orchestrator <Orchestrator> with instanceID <7c72ba4c-6459-4fae-84c2-23ba242ceda4>...
Instance started
Updating orchestrator state in table storage...
Orchestrator state updated
Any ideas why it might never leave pending status?
I recently had a feral orchestration persist in a retry and fail state for an extended period of time. Might I need to do some clean up in the dtfx control mechanisms in storage?
This application is running in the Central US region.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 26 (4 by maintainers)
Thanks for the information! I was able to reproduce this - the functions runtime host was recently updated to use Newtonsoft.Json 11.0.2, while one of the Durable Functions extension’s dependencies was still on 10.0.3. There’s a breaking change between the two versions that was causing an exception to be thrown whenever the Durable Task Framework tried to process a work item, which is why orchestrators were stuck in Pending state.
I’ve submitted a fix which will be going out in a release. In the meantime, please use Chris’s workaround.
@pluralmonad Unfortunately I think your original issue may have been overtaken by a similar issue which only impacted Functions v2. We’ve fixed the v2 issue, and several people are watching this issue to track that. It will be available in the 1.5 release, which is on nuget.org now (release announcement pending). For that reason, I’d like to close this issue. Can you open a new issue with your latest information, and specify that this is impacting Functions v1? Sorry for the mixup…
Thanks for confirming. This may have been caused by the recent update of the Azure Functions v2 runtime into Azure mentioned here.
Can you please try setting the
FUNCTIONS_EXTENSION_VERSIONapp setting to2.0.11776-alphato see if that mitigates the issue? This is a temporary way to rollback the runtime update for your function app.