azure-functions-durable-extension: Message loss due to race conditions with ContinueAsNew

Summary

Message loss or instance termination may be observed if an orchestrator completes after calling ContinueAsNew and subsequently processes any other message before restarting.

Details

When the orchestrator completes execution after ContinueAsNew is called, the status is internally marked as “completed” and a new message is enqueued to restart it. During this window between completion and restart, it’s possible for other messages to arrive (for example, raised events or termination messages). Because the internal state of the orchestration is completed, those messages will be dropped. It’s also possible for the DTFx runtime to terminate the instance claiming possible state corruption.

Repro

1. Start with the counter sample
2. Create a new instance
3. Call RaiseEventAsync(“operation”, “incr”) multiple times without waiting

Expected: Calling ContinueAsNew multiple times in quick succession should never be an issue. Many workloads may require this, especially actor-style workloads.

Workaround: Have the client wait a few seconds before sending events that may cause the orchestrator to do a ContinueAsNew. This give the instance time to get a new execution ID.