azure-functions-durable-extension: Function Apps using Durable Task in Isolated worker: Inner exception being lost
Description
After migrating to the Isolated worker process, we’ve encountered an issue with handling exceptions. The FunctionFailedException is no longer accessible in the latest worker packages. Despite extensive searching, there’s no clear documentation or online resources addressing this issue.
The documentation states that exceptions thrown in activity functions should be marshaled back to the orchestrator function as FunctionFailedException. However, in our case, they are propagating up as TaskFailedException’s. This behavior differs from the expected outcome, as outlined here: #https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=csharp-isolated#errors-in-activity-functions
Any exception that is thrown in an activity function is marshaled back to the orchestrator function and thrown as a
FunctionFailedException. You can write error handling and compensation code that suits your needs in the orchestrator function.
Expected behavior
The exception thrown contains the root exception and we have access to the FunctionFailedException
Actual behavior
The exception thrown contains only the message of the root exception and is a TaskFailedException
Relevant source code snippets
Snippet in debug mode:
Stacktrace:
Microsoft.DurableTask.TaskFailedException: Task 'GetCustomerIntegrationStatusActivity' (#6) failed with an unhandled exception: One or more errors occurred. (HealthCheckFailed Xero API health check failed. This could be transient on our side, transient on their side, or a more serious problem.)
---> DurableTask.Core.Exceptions.TaskFailedException: Exception of type 'DurableTask.Core.Exceptions.TaskFailedException' was thrown.
at DurableTask.Core.TaskOrchestrationContext.ScheduleTaskInternal(String name, String version, String taskList, Type resultType, Object[] parameters) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 118
at DurableTask.Core.TaskOrchestrationContext.ScheduleTaskToWorker[TResult](String name, String version, String taskList, Object[] parameters) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 89
at DurableTask.Core.TaskOrchestrationContext.ScheduleTask[TResult](String name, String version, Object[] parameters) in /_/src/DurableTask.Core/TaskOrchestrationContext.cs:line 81
at DurableTask.Core.RetryInterceptor`1.Invoke() in /_/src/DurableTask.Core/RetryInterceptor.cs:line 61
at DurableTask.Core.RetryInterceptor`1.Invoke() in /_/src/DurableTask.Core/RetryInterceptor.cs:line 96
at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.CallActivityAsync[T](TaskName name, Object input, TaskOptions options)
--- End of inner exception stack trace ---
at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.CallActivityAsync[T](TaskName name, Object input, TaskOptions options)
at Rifleman.Fintegration.Functions.Helpers.SyncExecutors.FullSyncExecutor.GetCustomerSyncStatus() in C:\Work\Rifleman.Fintegration\Rifleman.Fintegration\Rifleman.Fintegration.Functions\Helpers\SyncExecutors\FullSyncExecutor.cs:line 95
at Rifleman.Fintegration.Functions.Helpers.SyncExecutors.FullSyncExecutor.GetCustomerSyncStatus() in C:\Work\Rifleman.Fintegration\Rifleman.Fintegration\Rifleman.Fintegration.Functions\Helpers\SyncExecutors\FullSyncExecutor.cs:line 129
at Rifleman.Fintegration.Functions.Helpers.SyncExecutors.SyncExecutorBase.PerformPreSyncCheck() in C:\Work\Rifleman.Fintegration\Rifleman.Fintegration\Rifleman.Fintegration.Functions\Helpers\SyncExecutors\SyncExecutorBase.cs:line 79
Known workarounds
To get around this, we caught TaskFailedException’s in our Orchestrators and used string comparisons against the ex.FailureDetails.ErrorMessage to handle certain expectations 😢
App Details
- Azure Functions runtime version: v4
- Programming language used: C#
<ItemGroup>
<PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="1.0.0"/>
<PackageReference Include="Microsoft.Azure.Cosmos.Table" Version="1.0.8"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.19.0"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.0"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.0.0"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.14.1"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.2.0"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.15.1"/>
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0"/>
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="11.2.3"/>
<PackageReference Include="Microsoft.Azure.Storage.Queue" Version="11.2.3"/>
<PackageReference Include="Microsoft.DurableTask.Generators" Version="1.0.0-preview.1"/>
<PackageReference Include="Microsoft.Extensions.Http" Version="6.0.0"/>
</ItemGroup>
Any guidance would be appreciated.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 21 (9 by maintainers)
I apologize for bringing this up again, but we recently upgraded the DurableTask package to version
1.1.2. Unfortunately, the issue we were encountering hasn’t been resolved.The error message we’re receiving is still categorized as
unknownwhich prevents us from understanding the root cause of the failure. Furthermore, the inner failure is null.What we are seeing:
Due to the unknown error type, we’re unable to leverage the recommended, more specific error handling strategies from this thread. Instead, we’re forced to rely on a string comparison against the
ErrorMessage.These errors aren’t caught and instead it falls through to the general Catch at the bottom:
Any ideas or suggestions would be greatly appreciated, thanks.