azure-functions-durable-extension: Return message of Durable Entity is lost after the first time

Description

Return message of Durable Entity is lost after the first time.

To replicate this behaviour:

  1. Send one message to the client function like this:
{
	"id" : 1,
	"orchestratorName": "TestOrchestrator",
	"sampleText": "Hello World 1"
}
  1. Wait for completion

  2. Send one or more new messages to the client function like this (do not change the orchestrator name):

{
	"id" : 2,
	"orchestratorName": "TestOrchestrator",
	"sampleText": "Hello World 2"
}
  1. The orchestrator won’t receive the second message

Logs reports this:

Function 'TestOrchestrator (Orchestrator)' saved a '802d29f8-2b26-5d60-9197-1d34253271f6' event to an in-memory queue. State: ExternalEventDropped. HubName: TestTaskHub. AppName: . SlotName: . ExtensionVersion: 2.1.0. SequenceNumber: 220.

Expected behavior

CallEntityAsync returns the message every time.

Actual behavior

CallEntityAsync returns the message only the first time.

Relevant source code snippets

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Rest;
using Newtonsoft.Json;

[assembly: FunctionsStartup(typeof(TestFunction.Startup))]

namespace TestFunction
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            if (builder == null) throw new ArgumentNullException(nameof(builder));
            builder.Services.AddLogging();
        }
    }

    public class ClientModel
    {
        public int Id { get; set; }
        public string OrchestratorName { get; set; }
        public string SampleText { get; set; }
    }

    public class OrchestratorStatus
    {
        public string SampleText { get; set; }
        public bool? Action1Completed { get; set; }
        public int? RandomNumber { get; set; }
        public bool? Action2Completed { get; set; }
    }

    public class ClientFunction
    {
        readonly ILogger _log;

        public ClientFunction(ILogger<ClientFunction> log)
        {
            _log = log ?? throw new ArgumentNullException(nameof(log));
        }

        [FunctionName("ClientFunction")]
        public async Task<HttpResponseMessage> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "POST", Route = "EnqueueValue")] HttpRequestMessage reqMessage,
            [DurableClient(TaskHub = "TestTaskHub")] IDurableClient starter)
        {
            var stringContent = reqMessage.Content.AsString();
            var model = JsonConvert.DeserializeObject<ClientModel>(stringContent);
            var instanceId = OrchestratorFunction.ComposeInstanceId(model.OrchestratorName);

            _log.LogInformation($"Client Start - {model.OrchestratorName}");

            var orchestratorStatus = await starter.GetStatusAsync(instanceId).ConfigureAwait(false);

            _log.LogInformation($"Orchestrator status: '{(orchestratorStatus?.RuntimeStatus.ToString() ?? "NULL")}'");

            var entityId = EntityQueueFunction.MakeEntityId(model.OrchestratorName);

            _log.LogInformation($"Queue Entity: '{entityId.EntityKey}'");

            await starter.SignalEntityAsync(
                    entityId: entityId,
                    operationName: EntityQueueFunction.Operations.Enqueue,
                    operationInput: stringContent,
                    taskHubName: "TestTaskHub").ConfigureAwait(false);

            if (orchestratorStatus == null)
            {
                _log.LogInformation($"Starting orchestrator with ID '{instanceId}'");
                await starter.StartNewAsync(OrchestratorFunction.FUNCTION_NAME, instanceId).ConfigureAwait(false);
            }
            else
            {
                if (orchestratorStatus.RuntimeStatus == OrchestrationRuntimeStatus.Failed)
                {
                    _log.LogInformation($"Rewinding orchestrator with ID '{instanceId}'");
                    await starter.RewindAsync(instanceId, "Restart orchestrator after error").ConfigureAwait(false);
                }

                if (orchestratorStatus.RuntimeStatus == OrchestrationRuntimeStatus.Pending)
                {
                    await Task.Delay(TimeSpan.FromSeconds(60)).ConfigureAwait(false);
                    orchestratorStatus = await starter.GetStatusAsync(instanceId).ConfigureAwait(false);

                    if (orchestratorStatus.RuntimeStatus == OrchestrationRuntimeStatus.Pending)
                    {
                        _log.LogInformation($"Stopping orchestrator with ID '{instanceId}'");
                        await starter.TerminateAsync(instanceId, "Terminating pendind orchestrator").ConfigureAwait(false);
                    }

                    while ((orchestratorStatus = await starter.GetStatusAsync(instanceId).ConfigureAwait(false)).RuntimeStatus == OrchestrationRuntimeStatus.Pending)
                    {
                        await Task.Delay(TimeSpan.FromSeconds(10)).ConfigureAwait(false);
                    }
                }

                if (orchestratorStatus.RuntimeStatus == OrchestrationRuntimeStatus.Terminated)
                {
                    _log.LogInformation($"Restarting orchestrator with ID '{instanceId}'");
                    await starter.StartNewAsync(OrchestratorFunction.FUNCTION_NAME, instanceId).ConfigureAwait(false);
                }
            }

            return await starter.WaitForCompletionOrCreateCheckStatusResponseAsync(reqMessage, instanceId).ConfigureAwait(false);
        }
    }

    public class EntityQueueFunction
    {
        public static class Operations
        {
            public const string Enqueue = "Enqueue";
            public const string Dequeue = "Dequeue";
        }

        public const string ENTITY_NAME = nameof(TestFunction.EntityQueueFunction);

        private readonly ILogger _log;

        public EntityQueueFunction(ILogger<EntityQueueFunction> log)
        {
            _log = log ?? throw new ArgumentNullException(nameof(log));
        }

        public static EntityId MakeEntityId(string orchestratorName)
        {
            if (string.IsNullOrWhiteSpace(orchestratorName))
            {
                throw new ArgumentNullException(orchestratorName);
            }

            return new EntityId(ENTITY_NAME, $"Test.{orchestratorName.ToLowerInvariant()}");
        }

        [FunctionName(nameof(TestFunction.EntityQueueFunction))]
        public void EntityQueue([EntityTrigger]IDurableEntityContext ctx)
        {
            if (ctx == null) throw new ArgumentNullException(nameof(ctx));

            Queue<string> queue = ctx.GetState<Queue<string>>() ?? new Queue<string>();

            switch (ctx.OperationName ?? "")
            {
                case Operations.Enqueue:
                    try
                    {
                        _log.LogDebug($"Before Enqueue - Messenge in queue: {queue.Count}");
                        queue.Enqueue(ctx.GetInput<string>());
                    }
                    catch (Exception ex) when (ex is InvalidCastException || ex is JsonSerializationException)
                    {
                        queue = new Queue<string>();
                    }
                    finally
                    {
                        _log.LogDebug($"After Enqueue - Messenge in queue: {queue?.Count ?? -1}");
                        ctx.SetState(queue);
                    }
                    break;

                case Operations.Dequeue:
                    string message = "";
                    try
                    {
                        _log.LogDebug($"Before Dequeue - Messenge in queue: {queue.Count}");
                        if (queue.TryDequeue(out message))
                        {
                            ctx.SetState(queue);
                        }
                    }
                    catch (Exception ex) when (ex is InvalidCastException || ex is JsonSerializationException)
                    {
                        queue = new Queue<string>();
                        ctx.SetState(queue);
                    }
                    finally
                    {
                        _log.LogDebug($"After Dequeue - Messenge in queue: {queue?.Count ?? -1}");
                        ctx.Return(message);
                    }
                    break;
            }
        }
    }

    public class OrchestratorFunction
    {
        public const string FUNCTION_NAME = "TestOrchestrator";

        readonly ILogger _log;

        public OrchestratorFunction(ILogger<OrchestratorFunction> log)
        {
            _log = log ?? throw new ArgumentNullException(nameof(log));
        }

        [Deterministic]
        public static string ComposeInstanceId(string orchestratorName)
        {
            if (string.IsNullOrWhiteSpace(orchestratorName))
            {
                throw new ArgumentNullException(orchestratorName);
            }

            return $"Test.{FUNCTION_NAME}.{orchestratorName.ToLowerInvariant()}";
        }

        public static string GetOrchestratorName(string instanceId)
        {
            return instanceId?.Split('.')?.Last();
        }

        [FunctionName(FUNCTION_NAME)]
        public async Task RunAsync([OrchestrationTrigger]IDurableOrchestrationContext context)
        {
            if (context == null) throw new ArgumentNullException(nameof(context));

            var entityId = EntityQueueFunction.MakeEntityId(GetOrchestratorName(context.InstanceId));
            var input = await context.CallEntityAsync<string>(entityId, EntityQueueFunction.Operations.Dequeue);

            _log.LogDebug($"EntityId: {entityId} - IsReplay: {context.IsReplaying} - Input: {input ?? "NULL"}");

            if (!string.IsNullOrWhiteSpace(input))
            {
                var requestMessage = JsonConvert.DeserializeObject<ClientModel>(input);

                if (!context.IsReplaying)
                {
                    _log.LogInformation($"START - INSTANCE: {context.InstanceId ?? "NULL"}");
                    _log.LogInformation($"INPUT : {input}");
                }

                try
                {
                    var operationStatus = await context.CallSubOrchestratorAsync<OrchestratorStatus>(SubOrchestratorFunction.FUNCTION_NAME, new OrchestratorStatus
                    {
                        SampleText = requestMessage.SampleText
                    });

                    _log.LogInformation($"END WITH SUCCESS - INSTANCE {context.InstanceId}");

                }
                catch (Exception ex)
                {
                    _log.LogCritical(ex, $"END WITH ERROR - INSTANCE {context.InstanceId} - {ex.Message}");
                }
            }           

            context.ContinueAsNew(null, true);
        }
    }

    public class SubOrchestratorFunction
    {
        public const string FUNCTION_NAME = "TestSubOrchestrator";

        readonly ILogger _log;

        public SubOrchestratorFunction(ILogger<SubOrchestratorFunction> log)
        {
            _log = log ?? throw new ArgumentNullException(nameof(log));
        }

        [FunctionName(FUNCTION_NAME)]
        public async Task<OrchestratorStatus> RunAsync([OrchestrationTrigger]IDurableOrchestrationContext context)
        {
            if (context == null) throw new ArgumentNullException(nameof(context));

            var status = context.GetInput<OrchestratorStatus>();

            try
            {
                if (!status.Action1Completed.HasValue)
                {
                    if (!context.IsReplaying)
                    {
                        _log.LogDebug($"TestSubOrchestrator - Start - Action1Completed");
                    }

                    status.Action1Completed = await context.CallActivityAsync<bool>(TestActivitiyFunction1.FUNCTION_NAME, status.SampleText);

                    _log.LogDebug($"TestSubOrchestrator - End - Action1Completed");

                    context.ContinueAsNew(status, true);
                    return null;
                }

                if (!status.RandomNumber.HasValue)
                {
                    if (!context.IsReplaying)
                    {
                        _log.LogDebug($"TestSubOrchestrator - Start - RandomNumber");
                    }

                    status.RandomNumber = new Random().Next(1, 5);

                    _log.LogDebug($"TestSubOrchestrator - End - RandomNumber");

                    context.ContinueAsNew(status, true);
                    return null;
                }

                if (!status.Action2Completed.HasValue)
                {
                    if (!context.IsReplaying)
                    {
                        _log.LogDebug($"TestSubOrchestrator - Start - Action2Completed");
                    }

                    status.Action2Completed = await context.CallActivityAsync<bool>(TestActivitiyFunction2.FUNCTION_NAME, status.RandomNumber.Value);

                    _log.LogDebug($"TestSubOrchestrator - End - Action2Completed");

                    context.ContinueAsNew(status, true);
                    return null;
                }
            }
            catch (Exception ex) when (ex.InnerException is InvalidOperationException)
            {
                status.RandomNumber = null;
                context.ContinueAsNew(status, true);
                return null;
            }
            catch (Exception ex)
            {
                _log.LogCritical(ex, $"Error - {ex}");
                context.ContinueAsNew(status, true);
                return null;
            }

            if (!context.IsReplaying)
            {
                _log.LogDebug($"TestSubOrchestrator - END");
            }

            return status;
        }
    }

    public class TestActivitiyFunction1
    {
        public const string FUNCTION_NAME = "TestActivitiy1";

        readonly ILogger _log;

        public TestActivitiyFunction1(ILogger<TestActivitiyFunction1> log)
        {
            _log = log ?? throw new ArgumentNullException(nameof(log));
        }

        [FunctionName(FUNCTION_NAME)]
        public async Task<bool> VerifyAsync([ActivityTrigger]IDurableActivityContext context)
        {
            if (context == null) throw new ArgumentNullException(nameof(context));

            var input = context.GetInput<string>();

            _log.LogWarning($"Activity 1 input: {input}");

            await Task.Delay(1000);

            return true;
        }
    }

    public class TestActivitiyFunction2
    {
        public const string FUNCTION_NAME = "TestActivitiy2";

        [FunctionName(FUNCTION_NAME)]
        public async Task<bool> VerifyAsync([ActivityTrigger]IDurableActivityContext context)
        {
            if (context == null) throw new ArgumentNullException(nameof(context));

            var input = context.GetInput<int>();

            if (input % 2 == 1)
            {
                throw new InvalidOperationException("Input must be even");
            }

            await Task.Delay(input * 1000);

            return true;
        }
    }

}

App Details

  • Durable Functions extension version (e.g. v1.8.3): 2.1.1 (dev branch)
  • Azure Functions runtime version (1.0 or 2.0): 2.0
  • Programming language used: c#

Additional context

  • Development environment (ex. Visual Studio): Visual Studio 2019
  • Additional bindings used: HttpTrigger

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (4 by maintainers)

Most upvoted comments

Are there any appropriate workarounds for this in the meantime, given we’re losing messages on eternal orchestrations? It’s currently producing failures for our production environment.

I think this is caused by the message sorter logic, which is designed to filter duplicates and ensure in-order delivery of messages sent from orchestrators to entities. The problem here is that this logic was built with the assumption that each new orchestrator instance will have a fresh instance id. In this example, the old orchestrator instance id is reused which means the message sorter gets confused and does not deliver the message (because it thinks it is an out-of-order message).

A quick workaround would be to not re-use the orchestrator instance id.

Alternatively, to make the message sorting logic more robust, we could use a combination of (creation timestamp, instance id) instead of just the (instance id) to identify orchestrator instances. That will however mean we are using more space in each message and in the entity state.