azure-functions-durable-extension: Isolated Process with DurableClient and ASP.NET Core integration will throw exception on CreateCheckStatusResponse

Description Packages:

When using the version of the functions packages that provides ASP.NET Core integration and DurableTaskClient, the method CreateCheckStatusResponse will throw an InvalidOperationException because the implementation writes to the HttpResponseData synchronous

Expected behavior There should be no exception thrown, however as synchronous writing is not allowed since ASP.NET Core 3+ a Async version will be needed to be exposed.

Actual behavior System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead

Result: An unhandled exception occurred.
Exception: System.AggregateException: One or more errors occurred. (Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.)
  ---> System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
    at Azure.Core.Serialization.JsonObjectSerializer.Serialize(Stream stream, Object value, Type inputType, CancellationToken cancellationToken)
    at Microsoft.Azure.Functions.Worker.DurableTaskClientExtensions.CreateCheckStatusResponse(DurableTaskClient client, HttpRequestData request, String instanceId, HttpStatusCode statusCode, CancellationToken cancellation) in /_/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs:line 97
    at Microsoft.Azure.Functions.Worker.DurableTaskClientExtensions.CreateCheckStatusResponse(DurableTaskClient client, HttpRequestData request, String instanceId, CancellationToken cancellation) in /_/src/Worker.Extensions.DurableTask/DurableTaskClientExtensions.cs:line 34

Relevant source code snippets

[Function("SimpleHttpTriggerHttpData")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequestData req,
[DurableClient] DurableTaskClient durableClient)
{
       return durableClient.CreateCheckStatusResponse(req, "FakeId"); //Will result in exception
}

Known Workarounds The following configuration is possible, however this would be rather avoided

services.Configure<KestrelServerOptions>(options =>
{
    options.AllowSynchronousIO = true;
});

Other information When using ASP.NET Core integration it would be nice to also have a CreateCheckStatusResponse version that accepts HttpRequest and returns an ActionResult to keep it more consistent with the HttpTriggers.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (6 by maintainers)

Most upvoted comments

I would like to reopen this. I don’t think issue should be closed when default scafoldding for durable function in .NET Core Isolated project yields exeception and requires workarounds. This should be addressed. As a programmer I want the default setup to work. Seeing red error messages right after starting the sample project while following documentation is really not good user experience and if I was new to the technology I would be really questioning it

Create a new .NET 8 Durable Functions Orchestration project in VS 17.9.4. Then call Function1_HttpStart from a REST client. It will fail with the error described above.

Side-note: I can’t figure out a way to unit test Function1_HttpStart. An example would be great!

Honestly, you follow official documentation on how to create a new Durable Function which tells you to trigger the function through postman and you receive errors and status code 500, made me rethink what I am doing wrong and waste time on googling until I found this issue which is somehow closed.

Just confirming that as of 1.1.2, there’s a CreateCheckStatusResponseAsync that can be used. I believe the team is working on fixing the default templates to use the correct API.

Here’s what I got working after updating my nuget package references.

[Function("HelloCities_HttpStart")]
public static async Task<HttpResponseData> HttpStart(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
    [DurableClient] DurableTaskClient client,
    FunctionContext executionContext)
{
    ILogger logger = executionContext.GetLogger("HelloCities_HttpStart");

    // Function input comes from the request content.
    string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(nameof(HelloCities));

    logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

    // Returns an HTTP 202 response with an instance management payload.
    // See https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-http-api#start-orchestration
    return await client.CreateCheckStatusResponseAsync(req, instanceId);
}

I was able to get it to work following the instructions at this link: https://github.com/Azure/azure-functions-templates/pull/1513

But the final fix was to replace the whole line of code from this:

“return client.CreateCheckStatusResponse(req, instanceId);”

to this:

“return await client.CreateCheckStatusResponseAsync(req, instanceId);”

I added “await” and that got the code working.

Yeah we dropped the ball on this one, my apologies for not realizing the templates had stopped working; I’m looking to follow up with the tooling teams to understand the existing template validation flow did not catch this.

I think part of the confusion, on our end, was that it wasn’t fully clear (at least to me) that this exception was getting triggered with the templates. I’m not sure that this was the case when this GitHub issue first got created, but it is certainly the issue now.

In any case, I’ve fixed our templates here and will be following up with the VS and VSCode teams to make sure these get shipped asap. In the meantime, it might be good to create some kind of pinned comment flagging this issue. I don’t believe this thread is the right thread to pin or re-open, but instead we should create one talking specifically about templates. I’ll look to follow up - cc/ @lilyjma

We’ve been using function apps for about two years now. If I were evaluating them today, seeing the VS template fail and needing to pour through all this just to get a base example to work would have ruined my enthusiasm for Azure functions. Right now we’re migrating from .Net 6 to 8, and I’m seeing this issue. The recommended workaround is also not a recommended approach to asynchronous.

For anyone interested, here’s my simple and ingenious workaround.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.Logging;

namespace Testaroo
{
    public static class Durable2
    {
        [Function(nameof(Durable2))]
        public static async Task<List<string>> RunOrchestrator(
            [OrchestrationTrigger] TaskOrchestrationContext context)
        {
            ILogger logger = context.CreateReplaySafeLogger(nameof(Durable2));
            logger.LogInformation("Saying hello.");
            var outputs = new List<string>();

            // Replace name and input with values relevant for your Durable Functions Activity
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Tokyo"));
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "Seattle"));
            outputs.Add(await context.CallActivityAsync<string>(nameof(SayHello), "London"));

            // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
            return outputs;
        }

        [Function(nameof(SayHello))]
        public static string SayHello([ActivityTrigger] string name, FunctionContext executionContext)
        {
            ILogger logger = executionContext.GetLogger("SayHello");
            logger.LogInformation("Saying hello to {name}.", name);
            return $"Hello {name}!";
        }

        [Function("Durable2_HttpStart")]
        public static async Task<string> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
            [DurableClient] DurableTaskClient client,
            FunctionContext executionContext)
        {
            ILogger logger = executionContext.GetLogger("Durable2_HttpStart");

            // Function input comes from the request content.
            string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
                nameof(Durable2));

            logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

            // Returns an HTTP 202 response with an instance management payload.
            // See https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-http-api#start-orchestration
            //return client.CreateCheckStatusResponse(req, instanceId);

            return "WTF";
        }
    }
}

Hi @MarekLani - noted on the feedback, I agree the default set up should work; but I’m trying to make sure we’re talking about the same “default” set up.

Can you please help me understand what’s the sample project you’re using that’s leading to this “out of the box” exception? Some links to the docs or samples that are yielding this would help. Thanks!

@prashantagrawalcrowe There are three work arounds for now:

  1. (Recommended) When using ASP.NET Core integration adjust your azure function so that it enables synchronous writing. Just put the following snippet in the section where you configure your services in the Program.cs when configuring your host. I would recommend for now this unless it becomes too heavy and causes side effects. (See ASP.NET Core Kestrel synchronous-io: )
services.Configure<KestrelServerOptions>(options =>
{
    options.AllowSynchronousIO = true;
});
  1. For HttpTriggers with ASP.NET Integration that use HttpRequestData/HttpResponseData. The following copied version from the package source will work with the first work-around, however keep in mind that this will need to do reflection at a point due to the need to access an internal property. So you will need to keep an eye out on this repo all the time to ensure the reflection will work and if any changes have been made.
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.

using System;
using System.Net;
using System.Threading;
using Azure.Core.Serialization;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;

namespace Microsoft.Azure.Functions.Worker;

/// <summary>
/// Extensions for <see cref="DurableTaskClient"/>
/// </summary>
public static class DurableTaskClientExtensions
{
    /// <summary>
    /// Creates an HTTP response that is useful for checking the status of the specified instance.
    /// </summary>
    /// <param name="client">The <see cref="DurableTaskClient"/>.</param>
    /// <param name="request">The HTTP request that this response is for.</param>
    /// <param name="instanceId">The ID of the orchestration instance to check.</param>
    /// <param name="cancellation">The cancellation token.</param>
    /// <returns>An HTTP 202 response with a Location header and a payload containing instance control URLs.</returns>
    public static Task<HttpResponseData> CreateCheckStatusResponse(
        this DurableTaskClient client,
        HttpRequestData request,
        string instanceId,
        CancellationToken cancellation = default)
    {
        return client.CreateCheckStatusResponseAsync(request, instanceId, HttpStatusCode.Accepted, cancellation);
    }

    /// <summary>
    /// Creates an HTTP response that is useful for checking the status of the specified instance.
    /// </summary>
    /// <param name="client">The <see cref="DurableTaskClient"/>.</param>
    /// <param name="request">The HTTP request that this response is for.</param>
    /// <param name="instanceId">The ID of the orchestration instance to check.</param>
    /// <param name="statusCode">The status code.</param>
    /// <param name="cancellation">The cancellation token.</param>
    /// <returns>An HTTP response with a Location header and a payload containing instance control URLs.</returns>
    public static async Task<HttpResponseData> CreateCheckStatusResponseAsync(
        this DurableTaskClient client,
        HttpRequestData request,
        string instanceId,
        HttpStatusCode statusCode,
        CancellationToken cancellation = default)
    {
        if (client is null)
        {
            throw new ArgumentNullException(nameof(client));
        }

        static string BuildUrl(string url, params string?[] queryValues)
        {
            bool appended = false;
            foreach (string? query in queryValues)
            {
                if (!string.IsNullOrEmpty(query))
                {
                    url = url + (appended ? "&" : "?") + query;
                    appended = true;
                }
            }

            return url;
        }

        // TODO: To better support scenarios involving proxies or application gateways, this
        //       code should take the X-Forwarded-Host, X-Forwarded-Proto, and Forwarded HTTP
        //       request headers into consideration and generate the base URL accordingly.
        //       More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded.
        //       One potential workaround is to set ASPNETCORE_FORWARDEDHEADERS_ENABLED to true.
        string baseUrl = request.Url.GetLeftPart(UriPartial.Authority);
        string formattedInstanceId = Uri.EscapeDataString(instanceId);
        string instanceUrl = $"{baseUrl}/runtime/webhooks/durabletask/instances/{formattedInstanceId}";
        string? commonQueryParameters = GetQueryParams(client);

        HttpResponseData response = request.CreateResponse(statusCode);
        response.Headers.Add("Location", BuildUrl(instanceUrl, commonQueryParameters));
        response.Headers.Add("Content-Type", "application/json");

        ObjectSerializer serializer = GetObjectSerializer(response);
        var payload = new
        {
            id = instanceId,
            purgeHistoryDeleteUri = BuildUrl(instanceUrl, commonQueryParameters),
            sendEventPostUri = BuildUrl($"{instanceUrl}/raiseEvent/{{eventName}}", commonQueryParameters),
            statusQueryGetUri = BuildUrl(instanceUrl, commonQueryParameters),
            terminatePostUri = BuildUrl($"{instanceUrl}/terminate", "reason={{text}}}", commonQueryParameters),
        };

        // Need to capture context!
        await serializer.SerializeAsync(response.Body, payload, payload.GetType(), cancellation).ConfigureAwait(true);
        return response;
    }

    private static ObjectSerializer GetObjectSerializer(HttpResponseData response)
    {
        return response.FunctionContext.InstanceServices.GetService<IOptions<WorkerOptions>>()?.Value?.Serializer
            ?? throw new InvalidOperationException("A serializer is not configured for the worker.");
    }

    private static string? GetQueryParams(DurableTaskClient client)
    {
        // Note: Fragile, can break when you update your Microsoft.Azure.Functions.Worker.Extensions.DurableTask package!
        // Verify the source first before updating package if it will not break
        // https://github.com/Azure/azure-functions-durable-extension/blob/dev/src/Worker.Extensions.DurableTask/FunctionsDurableTaskClient.cs
        var clientType = client.GetType();
        if (client.GetType().FullName == "Microsoft.Azure.Functions.Worker.FunctionsDurableTaskClient"
            && clientType.GetProperty("QueryString") is { } queryStringProp)
        {
            return queryStringProp.GetValue(client, null) as string;
        }

        return null;
    }
}
  1. For HttpTriggers with ASP.NET Integration that use HttpRequest and IActionResult. The same applies as for point 2. It still needs to do reflection due to the need to access an internal property. So you will need to keep an eye out on this repo all the time to ensure the reflection will work and if any changes have been made.
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Extensions;
using Microsoft.AspNetCore.Mvc;
using Microsoft.DurableTask.Client;
using System.Net;

namespace Microsoft.Azure.Functions.Worker;

/// <summary>
/// Extensions for <see cref="DurableTaskClient"/>
/// </summary>
public static class DurableTaskClientExtensions
{
    /// <summary>
    /// Creates an <see cref="IActionResult"/> response that is useful for checking the status of the specified instance.
    /// </summary>
    /// <param name="client">The <see cref="DurableTaskClient"/>.</param>
    /// <param name="request">The HTTP request that this response is for.</param>
    /// <param name="instanceId">The ID of the orchestration instance to check.</param>
    /// <returns>An <see cref="IActionResult"/> with HTTP 202 response and a payload containing instance control URLs, to the <see cref="HttpResponse"/> the Location header is added.</returns>
    public static IActionResult CreateCheckStatusResponse(
        this DurableTaskClient client,
        HttpRequest request,
        string instanceId)
    {
        return client.CreateCheckStatusResponse(request, instanceId, HttpStatusCode.Accepted);
    }

    /// <summary>
    /// Creates an <see cref="IActionResult"/> response that is useful for checking the status of the specified instance.
    /// </summary>
    /// <param name="client">The <see cref="DurableTaskClient"/>.</param>
    /// <param name="request">The HTTP request that this response is for.</param>
    /// <param name="instanceId">The ID of the orchestration instance to check.</param>
    /// <param name="statusCode">The status code.</param>
    /// <returns>An <see cref="IActionResult"/> with the desired <paramref name="statusCode"/> a payload containing instance control URLs, to the <see cref="HttpResponse"/> the Location header is added.</returns>
    public static IActionResult CreateCheckStatusResponse(
        this DurableTaskClient client,
        HttpRequest request,
        string instanceId,
        HttpStatusCode statusCode)
    {
        if (client is null)
        {
            throw new ArgumentNullException(nameof(client));
        }

        static string BuildUrl(string url, params string?[] queryValues)
        {
            bool appended = false;
            foreach (string? query in queryValues)
            {
                if (!string.IsNullOrEmpty(query))
                {
                    url = url + (appended ? "&" : "?") + query;
                    appended = true;
                }
            }

            return url;
        }

        // TODO: To better support scenarios involving proxies or application gateways, this
        //       code should take the X-Forwarded-Host, X-Forwarded-Proto, and Forwarded HTTP
        //       request headers into consideration and generate the base URL accordingly.
        //       More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded.
        //       One potential workaround is to set ASPNETCORE_FORWARDEDHEADERS_ENABLED to true.
        string baseUrl = new Uri(request.GetEncodedUrl()).GetLeftPart(UriPartial.Authority);
        string formattedInstanceId = Uri.EscapeDataString(instanceId);
        string instanceUrl = $"{baseUrl}/runtime/webhooks/durabletask/instances/{formattedInstanceId}";
        string? commonQueryParameters = GetQueryParams(client);

        request.HttpContext.Response.Headers.Append("Location", BuildUrl(instanceUrl, commonQueryParameters));
        request.HttpContext.Response.Headers.Append("Content-Type", "application/json");

        return new ObjectResult(new
        {
            id = instanceId,
            purgeHistoryDeleteUri = BuildUrl(instanceUrl, commonQueryParameters),
            sendEventPostUri = BuildUrl($"{instanceUrl}/raiseEvent/{{eventName}}", commonQueryParameters),
            statusQueryGetUri = BuildUrl(instanceUrl, commonQueryParameters),
            terminatePostUri = BuildUrl($"{instanceUrl}/terminate", "reason={{text}}}", commonQueryParameters),
        })
        {
            StatusCode = (int)statusCode
        };
    }

    private static string? GetQueryParams(DurableTaskClient client)
    {
        // Note: Fragile, can break when you update your Microsoft.Azure.Functions.Worker.Extensions.DurableTask package!
        // Verify the source first before updating package if it will not break
        // https://github.com/Azure/azure-functions-durable-extension/blob/dev/src/Worker.Extensions.DurableTask/FunctionsDurableTaskClient.cs
        var clientType = client.GetType();
        if (client.GetType().FullName == "Microsoft.Azure.Functions.Worker.FunctionsDurableTaskClient"
            && clientType.GetProperty("QueryString") is { } queryStringProp)
        {
            return queryStringProp.GetValue(client, null) as string;
        }

        return null;
    }
}