azure-functions-durable-extension: Isolated Worker: Application Insights Durable Function Distributed Tracing not working
I have a .NET 7 Function App (Isolated Worker) that has Application Insights setup using the same instructions documented here. This app is also using Durable Functions.
Application Insights appears to be working just fine for non-durable functions as one would expect given this announcement. Unfortunately, I can’t seem to get distributed tracing to work for Durable Functions like it does for In-Process Durable Functions. After reading a lot of the existing issues here in this repo, it’s not clear to me whether it’s not fully supported yet or I’ve got something misconfigured in the app.
The closest I’ve been able to get is by following the README in the Distributed Tracing V2 for Durable Functions which looks to be targeted at In-Process Durable Functions. By updating the host.json config as documented here, I’ve been able to see Orchestrator & Activity Functions correlated together albeit with an Operation Name value of <Empty>. Furthermore, the ServiceBusTrigger Function that started the orchestrator is still not being correlated with the Orchestrator or Activities.
It’s also worth noting that when I took the same project and migrated it to In-Process Functions, the distributed tracing worked in the same way it’s documented here in the sample.
Any guidance from Microsoft or the community would be much appreciated.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 8
- Comments: 23 (5 by maintainers)
@nytian @bachuv It is now March - any updates?
@harleyz being unable set those values in dotnet isolated is expected right now due to the design. It is a different process entirely (the host) that is actually emitting the spans, so any modifications done in the worker process will not be reflected in the host process. This includes any
ITelemetryInitializerorITelemetryProcessoryou have in the worker - they will never see the emitted span telemetry because it is done in a different process.This is a known gap we will have in our Functions OpenTelemetry support we are adding. We will need to review this scenario and see if we can provide a way to flow back mutations on
Activity.Currentback to the host process.@bachuv and @RohitRanjanMS for awareness.
Thank you for tracking this, @eCorke Distributed tracing v2 is scheduled to be released on February. @bachuv can you help provide more details about this part?
Hi! I’m having the same issue. The activities are being correlated but they can be seen as
<Empty>in the Operation Name when using the app insights performance tab. The orchestration I’m testing is started by a http trigger and the start of the orchestration is not being correlated with the suborchestrations, activities, etc… I’m using .NET 8 in Isolated Worker…@cgillum Speaking for myself, I’m still seeing the behavior that I originally described when creating this issue.
@cgillum - I am getting the same behaviour as @RobARichardson . It is a little confusing though and maybe the docs just need to be improved. For example below is a request for faninout (shows as request in app insights). I cannot see the activity functions
When I do select a trace item though I get something like this
Also do I need to explicity call ConfigureFunctionsApplicationInsights I also see the empty operation name
Not much to add, but also have the same issue as others in this thread. dotnet 8 c# durable function isolated with distributed v2 tracing turned on and getting the
<empty>operation name and also not able to set cloud rolename.To preface, the distributed tracing as it is for durable functions is insanely useful especially when using complex patterns so I appreciate the effort being made here. The others in the thread above have touched on the issues I’m facing, but I’m just going to share my feedback.
Issue
I’m currently grappling with an issue regarding distributed tracing across different components of our architecture, particularly between a Function App utilizing Azure Durable Functions, Queue Storage, and an App Service instance. We have a setup where a client function, responsible for starting an orchestration, is triggered by a queue binding. This setup is supposed to facilitate automatic tracking.
Although tracing picks up the event of reading a message from the queue, this interaction doesn’t integrate as expected into the Application Map as expected. I followed the same setup as @RobARichardson , and while we can track the internal workings of the orchestrator and the activity functions it invokes, the tracing falls short in capturing the full path of the queue binding interaction externally which is a major challenge.
This image shows the client function (
QueueTriggeredExtractionOrchestrator) and the orchestrator function / activity functions (all showing up as<Empty>)Examining the client function, we see which queue the message came from, but no extended trace from the queue. I expected the dependency should show information from that. Additionally, it doesn’t show the distributed trace when the client function calls the orchestrator function.
In the orchestrator, the stack trace between the activity functions and orchestrator is good. I just can’t seem to figure out what information implies the relationship between the orchestrator and the client function
I also took a look at the referenced issue thread, but I wasn’t able to make use of the advice shared over there i.e. I don’t know what I’m looking for to access
Activity.Current.Related Issues
To further this point, when I have an activity function in the orchestrator call another sub-orchestrator via sending to a message that the other orchestrator relies on, the distributed trace is lost. For example, if I were to set the logging category to
Error, and my sub-orchestrator fails as a result of some flaw in the message, I would have 0 information on how the message was actually formed from the primary orchestrator since it’s not logged due to the logging category, and if I were to set it atInformation, it’s generally challenging to isolate the relevant traces. Any advice here would be appreciated anyways since I’m newer to Application Insights.On a related note, our backend App Service, which sends messages to this same queue, shows successful connections on the Application Map and I can access the distributed trace. Ideally, I’d like to see the distributed trace linking the function app all the way back to the web app, especially when errors are raised. However, there seems to be a disconnect in the tracing from the web app to the function app, despite both components interacting with the same queue storage.
Dont know if this is related, but I have a similar issue, where the Monitor view in Azure Portal does not show any Orchestration Traces for an orchestration trigger.
Seems like the following three properties are missing from customDimentions: DurableFunctionsInstanceId, DurableFunctionsRuntimeStatus and DurableFunctionsType
In-process function
Isolated worker function
Hi @RobARichardson, thanks for following up! I’ll try to repro and investigate the issue so we can find a fix.
Hi, @RobARichardson sorry I thought your question was answered. Tag @bachuv who works for distributed tracing for further answers. Thanks!
This attribute: https://github.com/Azure/azure-functions-durable-extension/blob/b5c70e5e40d12a1aed281ada1c4d5651608df247/src/Worker.Extensions.DurableTask/AssemblyInfo.cs#L7
Instructs the function SDK to fetch the DurableTask WebJobs nuget as a host extension. So, it is referenced, just through a functions specific way as these assemblies are needed for the host process only - not your own.
@RobARichardson only distributed tracing v2 is supported for dotnet isolated. This feature is still in preview and we are exploring ways to improve it. The
ServiceBusTrigger(or rather, the parent span) not being correlated would be a feature gap we have and will look into addressing.Will also look into operation name being empty.