azure-functions-durable-extension: Functions always fails with InvalidCastException when distributed tracing is enabled

Description

When our durable function is invoked it always fails with System.InvalidCastException. During development on the local machine, it works fine, and we get the desired End-to-End tracing on AppInsights. It is related to #1611 opening this new ticket as that was closed with comment asking to do so.

Expected behavior

It should run without exception and collect information to be able to display End-to-End tracing through all Activities called from the Orchestrator.

Actual behavior

It’s working as expected when we run it locally with functions runtime:

Core Tools Version:       3.0.3568 Commit hash: e30a0ede85fd498199c28ad699ab2548593f759b  (64-bit)
Function Runtime Version: 3.0.15828.0

On Azure it fails with the following exception:

Exception while executing function: ProcessItemRequests Unable to cast object of type 'DurableTask.Core.W3CTraceContext' to type 'DurableTask.Core.TraceContextBase'.

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:326)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__15.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:94)
Inner exception System.InvalidCastException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at DurableTask.Core.CorrelationTraceClient+<>c__DisplayClass7_0.<SetUp>b__1 (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.cs:60)
   at System.Reactive.AnonymousObserver`1.OnNextCore (System.Reactive, Version=4.4.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Rx.NET/Source/src/System.Reactive/AnonymousObserver.cs:67)
   at System.Reactive.ObserverBase`1.OnNext (System.Reactive, Version=4.4.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Rx.NET/Source/src/System.Reactive/ObserverBase.cs:34)
   at System.Diagnostics.DiagnosticListener.Write (System.Diagnostics.DiagnosticSource, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at DurableTask.Core.CorrelationTraceClient+<>c__DisplayClass9_0.<TrackDepencencyTelemetry>b__0 (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.cs:95)
   at DurableTask.Core.CorrelationTraceClient.Execute (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.cs:144)
   at DurableTask.Core.CorrelationTraceClient.TrackDepencencyTelemetry (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.cs:95)
   at DurableTask.Core.TaskHubClient.CreateAndTrackDependencyTelemetry (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\TaskHubClient.cs:669)
   at DurableTask.Core.TaskHubClient+<>c__DisplayClass27_0.<InternalCreateOrchestrationInstanceWithRaisedEventAsync>b__1 (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\TaskHubClient.cs:611)
   at DurableTask.Core.CorrelationTraceClient.Execute (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.cs:144)
   at DurableTask.Core.TaskHubClient+<InternalCreateOrchestrationInstanceWithRaisedEventAsync>d__27.MoveNext (DurableTask.Core, Version=2.5.2.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\TaskHubClient.cs:611)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient+<Microsoft-Azure-WebJobs-Extensions-DurableTask-IDurableOrchestrationClient-StartNewAsync>d__31`1.MoveNext (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.cs:170)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Func.Npr.ContractRules.Engine.Functions.KafkaTriggerFunction+<ProcessItemRequests>d__1.MoveNext (Func.Npr.ContractRules.Engine, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\BldAgntTrisus\Agent-A-build07-1\_work\312\s\Func.Npr.ContractRules.Engine\Functions\KafkaTriggerFunction.cs:41)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:20)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:555)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:501)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:279)

Relevant source code snippets

host.json

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "hubName": "%NprTaskHub%" ,
      "maxConcurrentOrchestratorFunctions": 10,
      "maxConcurrentActivityFunctions": 10,
      "tracing": {
        "distributedTracingEnabled": true,
        "distributedTracingProtocol": "W3CTraceContext"
      }
    },
    "kafka": {
      "maxBatchSize": 3,
      "MaxPollIntervalMs": 30000
    }
  },
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      },
      "httpAutoCollectionOptions": {
        "enableW3CDistributedTracing": true
      }
    }
  }
}

Known workarounds

Disable distributed tracing.

App Details

  • Durable Functions extension version (e.g. v1.8.3): 2.5.1
  • Azure Functions runtime version (1.0 or 2.0): 3.1.3 Commit hash: 9a07f8b0506c207ce547e275ee4e735785639ffa
  • Programming language used: .NET Core/C#

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 21 (2 by maintainers)

Most upvoted comments

@cgillum It seems that last one did the trick. Wee need a bit more time to test it, but seems promising so far.

UPDATE: While it not fails and collects most of the tracing required for End-to-End view, there are some issues still. Not sure if it has been reported already or it’s intentional, but as Actions and Orchestrators are evaluated multiple times until they finish on await, we get multiple logs for them, but their Parent Ids are not in sync.

Anyways the problem this ticket was opened for is fixed, so the issue can be closed. Thanks for the help!

+1
gcsepregi on Sep 3, 2021

@cgillum That’s what I’m just about to try. I’ve already made the change, just waiting to get through the CI/CD pipeline. I just remembered a couple of years ago solving similar issue this way just referencing the package directly. I’ll get back to you as soon as I run the tests.

+1
gcsepregi on Sep 2, 2021

@gcsepregi which version of the Durable Functions extension are you using (the Microsoft.Azure.WebJobs.Extensions.DurableTask nuget package)?

Also adding @bachuv and @TsuyoshiUshio who have more context on the distributed tracing feature.

+1
cgillum on Aug 5, 2021
Read more comments on GitHub
← azure-functions-durable-extension: Functions always fails with InvalidCastException when distributed tracing is enabled
azure-functions-durable-extension: Host startup fails under certain conditions if both Durable Task and Storage extensions are referenced. →