azure-functions-durable-extension: Root cause exception is not exposed in durable function orchestrator
Description
This might be more of an oversight or missing for some other reason, anyways here it is:
Throwing an exception in an activity function causes a DurableTask.Core.Exceptions.TaskFailedException exception. I cannot find a way to extract the root exception from this exception type. Is it supposed to be there? All I get is a message and a type, but the type is an AggregateException.
Also, the documentation here https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=csharp-isolated#errors-in-activity-functions states that it should be a FunctionFailedException exception. When I used inprocess durable functions this exception was thrown and it was possible to extract the root exception. In isolated the documentation is wrong.
Expected behavior
The exception thrown contains the root exception
Actual behavior
The exception thrown containsonly the message of the root exception
App Details
- Durable Functions extension version (e.g. v1.8.3): “Microsoft.Azure.Functions.Worker.Extensions.DurableTask” Version=“1.0.0” “Microsoft.Azure.WebJobs.Extensions.DurableTask” Version=“2.9.2”
- Azure Functions runtime version (1.0 or 2.0): 2
- Programming language used: dotnet
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 11
- Comments: 18 (7 by maintainers)
Commits related to this issue
- Log exceptions from activities Because of an issue with the isolated worker the root exceptions from activities are not exposed. Therefor we explicitly capture and log the exceptions before they bubb... — committed to WeAreInSpark/key-rotation-bot by sschutten 3 years ago
- Log exceptions from activities (#16) Because of an issue with the isolated worker the root exceptions from activities are not exposed. Therefor we explicitly capture and log the exceptions before the... — committed to WeAreInSpark/key-rotation-bot by sschutten 3 years ago
Same issue here I think using DurableTask 1.1.7. My Activity function is throwing a custom exception type which contains crucial state for handling the exception, but the exception caught in the orchestrator is an AggregateException with only one InnerException (regardless of how many retries failed) and the InnerException is a Microsoft.DurableTask.TaskFailedException with an inner DurableTask.Core.Exceptions.TaskFailedException. The result is after retries have been exhausted, I can’t get the detailed success/failure object returned by my Activity Function, which I had set on a property of a custom exception type to accommodate orchestrator retries. I.e., there’s no way to pass back information about the original exception or cause of the failure for final exception handling.
I’m working on some fixes which will address the missing inner exception problem. Stay tuned for more updates.
After migrating to the Isolated worker process, we’ve encountered the same issue with handling exceptions. The
FunctionFailedExceptionis no longer accessible in the latest worker packages. Despite extensive searching, there’s no clear documentation or online resources addressing this issue.The documentation states that exceptions thrown in activity functions should be marshaled back to the orchestrator function as
FunctionFailedException. However, in our case, they are propagating up asTaskFailedException. This behavior differs from the expected outcome, as outlined in this open ticket.We kindly request an update on the progress towards addressing this issue. The inability to utilize
FunctionFailedExceptionis hindering our ability to effectively manage exceptions in the Isolated worker process. A prompt resolution would be greatly appreciated.Example of
TaskFailedExceptionsurfacingCode changes we had to make when moving to an Isolated worker.
@eholman @irhad-ljubcic various fixes have been made over the last few months, including a couple in the last week or so which will hopefully get released soon:
In my testing, custom retry handlers have been working fine, so we’ll need more details about what specifically isn’t working for you.
Bumping this after a chat with @cgillum at Build on 5/21. As mentioned above, custom exception types are losing their additional property information which prevents us from properly handling the caught exceptions. Thank you!
For
AggregateException, I believe this is the cause: https://github.com/Azure/azure-functions-dotnet-worker/issues/993.Will need to see if we can even work around this, given it is an issue with the dotnet worker and not durable.
I don’t recall
1.0.2having any fixes related to this (but I may be wrong). I think what @torepaulsson is touching on is that in dotnet-isolated we switched to a failure-details only mode, and explicitly do not attempt to serializing/deserialize the original exception across the boundary. We do have a TODO to populate nested exceptions, but I think I want to look into this experience a bit more, I am not entirely sold it was the right call to remove exception propagation across the serialization boundary.As for
AggregateException- are you calling anyTask.ResultorTask.WaitAPIs? Those will cause any exception to be surfaced asAggregateException.@torepaulsson does this issue occur if you upgrade to the latest version of
Microsoft.Azure.Functions.Worker.Extensions.DurableTask(v1.0.2)?FYI @jviau