azure-functions-durable-extension: Distributed tracing doesn't correlate when using System.Diagnostics.DiagnosticSource 4.7.1

Description

I’ve setup distributed tracing for Application Insights in my durable functions. The DtOrchestrator and DtActivity items are correlated correctly, but my Traces (LogInformation, LogDebug, LogTrace) are not correlated to the Activities anymore. I’m pretty sure this worked a while ago, and also in the early preview NuGet package. The traces are logged to AI, just not linked to any DtActivity or DtOrchestrator instance.

Expected behavior

I expect to see the Traces correlated to a specific DtActivity, so I can filter them by Activity. In this pane for example:

Actual behavior

The Trace messages are not correlated to the Activity, so I have to search for them separately to browse logs.

Relevant source code snippets

Logging setup has been tried with and without W3CDistributedTracing enabled e.g.:

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "Default": "Information",
      "DurableTask": "Warning",
      "Function": "Trace",
      "Host": "Warning",
      "Microsoft.Azure.WebJobs": "Warning"
    },
    "applicationInsights": {
      "httpAutoCollectionOptions": {
        "enableW3CDistributedTracing": true
      },
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    }
  },
  "extensions": {
    "durableTask": {
      "hubName": "WorkspacesHub",
      "tracing": {
        "distributedTracingEnabled": true,
        "distributedTracingProtocol": "W3CTraceContext"
      }
    }
  }
}

Known workarounds

(None)

App Details

Durable Functions extension version: v2.4.0
Azure Functions runtime version: v3
Programming language used: C#

Screenshots

As an example, these are the properties returned for a DtActivity and should-be-correlated Trace message

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 32 (8 by maintainers)

Most upvoted comments

Hi @mauve. Yes, the update is still planned to be released around May 28th, most likely to be released a few days after that due to some extra validation steps we’re trying for the first time. You can track our release progress here: https://github.com/Azure/azure-functions-durable-extension/issues/1852 .

Relatedly, we always create release checklist-issues like the one above around our release dates, so looking for them is a good way to get up-to-date information on how the release is progressing 😃 .

@NickSevens, I’ll defer those questions to @bachuv, who should have more context on these.

davidmrdavid on May 27, 2021

Hi @davidmrdavid any news on this?

santo2 on Apr 14, 2021

It seems to me correlation now works correctly - when indeed upgrading to the latest packages and Azure Func V4. Which is good enough for me 😉

Thank you for the support guys.

NickSevens on Oct 17, 2022

@JakubMajsniar using net6 with version 4 of azure function runtime and system.diagnosticsource v6.0.0 did the trick tor me

dlebee on May 10, 2022

@mauve, could you please open up a new ticket describing this issue? Since this reproduction is for a different DiagnosticSource version, it might be best to separate the threads 😃 . Thank you!

davidmrdavid on Jun 4, 2021

Hi, any update? Or is there a workaround we can do?

mauve on May 26, 2021

Hi folks! Unfortunately we’ve been slowed down by prioritizing this major bug but I’ll take a different approach during our next release sprint and prioritize this instead. I think the bug I linked above is going to be solved on a much longer timeframe than I originally expected, so I can allow myself to multi-task on the issues moving forward 😃

Expect this to be worked on for our upcoming milestone. We haven’t gotten to the release planning stage yet (we just had a release) but I understand we should be starting that process soon.

davidmrdavid on Feb 1, 2021

Thanks @NickSevens!. I’ll need to dive into the documentation to understand why this would break our expected behavior. Thanks for providing this context, this is really helpful 😃

davidmrdavid on Dec 16, 2020

@davidmrdavid I’ve narrowed it down to this: I recently upgraded a NuGet package which has a dependency on System.Diagnostics.DiagnosticSource. The update caused the dependency to go from 4.6.0 to 4.7.1. This causes the correlation to fail.

When the correlation fails, I noticed a difference in Operation ID format:

Working correlation Operation ID has a format of for example 00-dc5421953c0c76468418424c05b840ce-70ae1d9822e11c44-00
Failing correlation Operation ID has a format of for example a61c9fdd-4739b89b1bfe50b0

As a reproduction, you can simply create a new project, and add the NuGet package. <PackageReference Include="System.Diagnostics.DiagnosticSource" Version="4.6.0" /> --> works <PackageReference Include="System.Diagnostics.DiagnosticSource" Version="4.7.1" /> --> fails

NickSevens on Dec 16, 2020

@mpaul31 , thanks. I’ll turn that into a separate ToDo item and prioritize it for an upcoming release of Dist. Tracing

davidmrdavid on Dec 16, 2020

Hi @mpaul31, you’re correct. The feature is still in public preview so the experience needs some more work before being ready for production. Just for my reference, where are you seeing this <Empty> operation name. Do you mind clarifying / sharing a screenshot Thanks!

davidmrdavid on Dec 16, 2020

Hi @davidmrdavid, I tried creating a minimal sample, but there it does correlate correctly. So … I’m now trying to figure out what’s different in my app. 😃 I’ll post here if I find anything causing the issue at my end…

NickSevens on Dec 16, 2020