azure-functions-durable-extension: Root cause exception is not exposed in durable function orchestrator

Description

This might be more of an oversight or missing for some other reason, anyways here it is:

Throwing an exception in an activity function causes a DurableTask.Core.Exceptions.TaskFailedException exception. I cannot find a way to extract the root exception from this exception type. Is it supposed to be there? All I get is a message and a type, but the type is an AggregateException.

Also, the documentation here https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=csharp-isolated#errors-in-activity-functions states that it should be a FunctionFailedException exception. When I used inprocess durable functions this exception was thrown and it was possible to extract the root exception. In isolated the documentation is wrong.

Expected behavior

The exception thrown contains the root exception

Actual behavior

The exception thrown containsonly the message of the root exception

App Details

  • Durable Functions extension version (e.g. v1.8.3): “Microsoft.Azure.Functions.Worker.Extensions.DurableTask” Version=“1.0.0” “Microsoft.Azure.WebJobs.Extensions.DurableTask” Version=“2.9.2”
  • Azure Functions runtime version (1.0 or 2.0): 2
  • Programming language used: dotnet

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 11
  • Comments: 18 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Same issue here I think using DurableTask 1.1.7. My Activity function is throwing a custom exception type which contains crucial state for handling the exception, but the exception caught in the orchestrator is an AggregateException with only one InnerException (regardless of how many retries failed) and the InnerException is a Microsoft.DurableTask.TaskFailedException with an inner DurableTask.Core.Exceptions.TaskFailedException. The result is after retries have been exhausted, I can’t get the detailed success/failure object returned by my Activity Function, which I had set on a property of a custom exception type to accommodate orchestrator retries. I.e., there’s no way to pass back information about the original exception or cause of the failure for final exception handling.

I’m working on some fixes which will address the missing inner exception problem. Stay tuned for more updates.

After migrating to the Isolated worker process, we’ve encountered the same issue with handling exceptions. The FunctionFailedException is no longer accessible in the latest worker packages. Despite extensive searching, there’s no clear documentation or online resources addressing this issue.

The documentation states that exceptions thrown in activity functions should be marshaled back to the orchestrator function as FunctionFailedException. However, in our case, they are propagating up as TaskFailedException. This behavior differs from the expected outcome, as outlined in this open ticket.

We kindly request an update on the progress towards addressing this issue. The inability to utilize FunctionFailedException is hindering our ability to effectively manage exceptions in the Isolated worker process. A prompt resolution would be greatly appreciated.

Example of TaskFailedException surfacing image

Code changes we had to make when moving to an Isolated worker. image

@eholman @irhad-ljubcic various fixes have been made over the last few months, including a couple in the last week or so which will hopefully get released soon:

Also. the RetryPolicy like the docs are mentioning aren’t working either?

In my testing, custom retry handlers have been working fine, so we’ll need more details about what specifically isn’t working for you.

Bumping this after a chat with @cgillum at Build on 5/21. As mentioned above, custom exception types are losing their additional property information which prevents us from properly handling the caught exceptions. Thank you!

For AggregateException, I believe this is the cause: https://github.com/Azure/azure-functions-dotnet-worker/issues/993.

Will need to see if we can even work around this, given it is an issue with the dotnet worker and not durable.

I don’t recall 1.0.2 having any fixes related to this (but I may be wrong). I think what @torepaulsson is touching on is that in dotnet-isolated we switched to a failure-details only mode, and explicitly do not attempt to serializing/deserialize the original exception across the boundary. We do have a TODO to populate nested exceptions, but I think I want to look into this experience a bit more, I am not entirely sold it was the right call to remove exception propagation across the serialization boundary.

As for AggregateException - are you calling any Task.Result or Task.Wait APIs? Those will cause any exception to be surfaced as AggregateException.

@torepaulsson does this issue occur if you upgrade to the latest version of Microsoft.Azure.Functions.Worker.Extensions.DurableTask (v1.0.2)?

FYI @jviau