azure-functions-durable-extension: Function Apps using Durable Task in Isolated worker: Inner exception being lost

Description

After migrating to the Isolated worker process, we’ve encountered an issue with handling exceptions. The FunctionFailedException is no longer accessible in the latest worker packages. Despite extensive searching, there’s no clear documentation or online resources addressing this issue.

The documentation states that exceptions thrown in activity functions should be marshaled back to the orchestrator function as FunctionFailedException. However, in our case, they are propagating up as TaskFailedException’s. This behavior differs from the expected outcome, as outlined here: #https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=csharp-isolated#errors-in-activity-functions

Any exception that is thrown in an activity function is marshaled back to the orchestrator function and thrown as a FunctionFailedException. You can write error handling and compensation code that suits your needs in the orchestrator function.

Expected behavior

The exception thrown contains the root exception and we have access to the FunctionFailedException

Actual behavior

The exception thrown contains only the message of the root exception and is a TaskFailedException

Relevant source code snippets

Snippet in debug mode:

image

Stacktrace:

Microsoft.DurableTask.TaskFailedException: Task 'GetCustomerIntegrationStatusActivity' (#6) failed with an unhandled exception: One or more errors occurred. (HealthCheckFailed Xero API health check failed. This could be transient on our side, transient on their side, or a more serious problem.)
 ---> DurableTask.Core.Exceptions.TaskFailedException: Exception of type 'DurableTask.Core.Exceptions.TaskFailedException' was thrown.
   at DurableTask.Core.TaskOrchestrationContext.ScheduleTaskInternal(String name, String version, String taskList, Type resultType, Object[] parameters) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 118
   at DurableTask.Core.TaskOrchestrationContext.ScheduleTaskToWorker[TResult](String name, String version, String taskList, Object[] parameters) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 89
   at DurableTask.Core.TaskOrchestrationContext.ScheduleTask[TResult](String name, String version, Object[] parameters) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 81
   at DurableTask.Core.RetryInterceptor`1.Invoke() in /_/src/DurableTask.Core/RetryInterceptor.cs:line 61
   at DurableTask.Core.RetryInterceptor`1.Invoke() in /_/src/DurableTask.Core/RetryInterceptor.cs:line 96
   at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.CallActivityAsync[T](TaskName name, Object input, TaskOptions options)
   --- End of inner exception stack trace ---
   at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.CallActivityAsync[T](TaskName name, Object input, TaskOptions options)
   at Rifleman.Fintegration.Functions.Helpers.SyncExecutors.FullSyncExecutor.GetCustomerSyncStatus() in C:\Work\Rifleman.Fintegration\Rifleman.Fintegration\Rifleman.Fintegration.Functions\Helpers\SyncExecutors\FullSyncExecutor.cs:line 95
   at Rifleman.Fintegration.Functions.Helpers.SyncExecutors.FullSyncExecutor.GetCustomerSyncStatus() in C:\Work\Rifleman.Fintegration\Rifleman.Fintegration\Rifleman.Fintegration.Functions\Helpers\SyncExecutors\FullSyncExecutor.cs:line 129
   at Rifleman.Fintegration.Functions.Helpers.SyncExecutors.SyncExecutorBase.PerformPreSyncCheck() in C:\Work\Rifleman.Fintegration\Rifleman.Fintegration\Rifleman.Fintegration.Functions\Helpers\SyncExecutors\SyncExecutorBase.cs:line 79

Known workarounds

To get around this, we caught TaskFailedException’s in our Orchestrators and used string comparisons against the ex.FailureDetails.ErrorMessage to handle certain expectations 😢

App Details

  • Azure Functions runtime version: v4
  • Programming language used: C#
<ItemGroup>
        <PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="1.0.0"/>
        <PackageReference Include="Microsoft.Azure.Cosmos.Table" Version="1.0.8"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.19.0"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.0"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.0.0"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.14.1"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.2.0"/>
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.15.1"/>
        <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0"/>
        <PackageReference Include="Microsoft.Azure.Storage.Blob" Version="11.2.3"/>
        <PackageReference Include="Microsoft.Azure.Storage.Queue" Version="11.2.3"/>
        <PackageReference Include="Microsoft.DurableTask.Generators" Version="1.0.0-preview.1"/>
        <PackageReference Include="Microsoft.Extensions.Http" Version="6.0.0"/>
</ItemGroup>

Any guidance would be appreciated.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 21 (9 by maintainers)

Most upvoted comments

I apologize for bringing this up again, but we recently upgraded the DurableTask package to version 1.1.2. Unfortunately, the issue we were encountering hasn’t been resolved.

The error message we’re receiving is still categorized as unknown which prevents us from understanding the root cause of the failure. Furthermore, the inner failure is null.

What we are seeing: image

Due to the unknown error type, we’re unable to leverage the recommended, more specific error handling strategies from this thread. Instead, we’re forced to rely on a string comparison against the ErrorMessage.

These errors aren’t caught and instead it falls through to the general Catch at the bottom: image

Any ideas or suggestions would be greatly appreciated, thanks.