azure-functions-durable-extension: Functions always fails with InvalidCastException when distributed tracing is enabled

Description

Adding distributedTracingEnabled = true to enable distributed tracing in existing Durable Functions-based applications will now fail with a runtime exception.

This seems to occur when newly enabling distributed tracing. Once the distributed tracing worked correctly, it did not occur thereafter.

Expected behavior

Execution completed without exceptions.

Actual behavior

Exception message

Exception while executing function: RenewCertificates_Timer Unable to cast object of type 'DurableTask.Core.HttpCorrelationProtocolTraceContext' to type 'DurableTask.Core.TraceContextBase'.

Stack trace

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 330)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__15.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 94)
Inner exception System.InvalidCastException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at DurableTask.Core.CorrelationTraceClient+<>c__DisplayClass7_0.<SetUp>b__1 (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 60)
   at System.Reactive.AnonymousObserver`1.OnNextCore (System.Reactive.Core, Version=3.0.6000.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263)
   at System.Reactive.ObserverBase`1.OnNext (System.Reactive.Core, Version=3.0.6000.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263)
   at System.Diagnostics.DiagnosticListener.Write (System.Diagnostics.DiagnosticSource, Version=4.0.5.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at DurableTask.Core.CorrelationTraceClient+<>c__DisplayClass9_0.<TrackDepencencyTelemetry>b__0 (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 95)
   at DurableTask.Core.CorrelationTraceClient.Execute (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 144)
   at DurableTask.Core.CorrelationTraceClient.TrackDepencencyTelemetry (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 95)
   at DurableTask.Core.TaskHubClient.CreateAndTrackDependencyTelemetry (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\TaskHubClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 669)
   at DurableTask.Core.TaskHubClient+<>c__DisplayClass27_0.<InternalCreateOrchestrationInstanceWithRaisedEventAsync>b__1 (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\TaskHubClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 611)
   at DurableTask.Core.CorrelationTraceClient.Execute (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\CorrelationTraceClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 144)
   at DurableTask.Core.TaskHubClient+<InternalCreateOrchestrationInstanceWithRaisedEventAsync>d__27.MoveNext (DurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89ddDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: C:\source\durabletask\src\DurableTask.Core\TaskHubClient.csDurableTask.Core, Version=2.5.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: 611)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient+<Microsoft-Azure-WebJobs-Extensions-DurableTask-IDurableOrchestrationClient-StartNewAsync>d__31`1.MoveNext (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.csMicrosoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: 170)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at AppService.Acmebot.Functions.RenewCertificates+<Timer>d__2.MoveNext (AppService.Acmebot, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullAppService.Acmebot, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\Users\shibayan\Documents\GitHub\appservice-acmebot\AppService.Acmebot\Functions\RenewCertificates.csAppService.Acmebot, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 123)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 20)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 559)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 505)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.25.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 283)

Relevant source code snippets

host.json

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request;Exception"
      }
    },
    "logLevel": {
      "KeyVault.Acmebot": "Warning"
    }
  },
  "extensions": {
    "durableTask": {
      "hubName": "DurableFunctionsHub",
      "extendedSessionsEnabled": true,
      "extendedSessionIdleTimeoutInSeconds": 120,
      "tracing": {
        "distributedTracingEnabled": true
      }
    },
    "http": {
      "customHeaders": {
        "Cache-Control": "no-store"
      }
    }
  }
}

Known workarounds

Disable distributed tracing.

App Details

  • Durable Functions extension version (e.g. v1.8.3): 2.4.0
  • Azure Functions runtime version (1.0 or 2.0): 3.0
  • Programming language used: .NET Core / C#

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (13 by maintainers)

Most upvoted comments

Hi @helsaawy,

Thanks for reaching out. Could you please open a new issue containing your durable-extension version as well as the functions host version you’re running on? That will help us kickstart an investigation 😃

cc/ @bachuv

+1
davidmrdavid on Jun 3, 2021

I tried again, and it appears to have worked correctly with the current latest Function Runtime Host (3.0.15406).

Error : 3.0.15193 Worked : 3.0.15406

I don’t keep track of what updates have been made during this version, but as long as no regressions occur, I don’t think it will be a problem. Thanks!

+1
shibayan on Mar 12, 2021

Hi @davidmrdavid ,

My apologies for the late response.

Yes, you’re right. In my case I’m using a “TimerTrigger” function as the “DurableClient” and the executions run without any issues each time the function is auto-triggered at its schedule (each 20 mins in my case). The problem comes when I run the function manually through the Azure Portal (because I need to start the process earlier for example). In that case, the error is the one I pasted in my previous message.

Because you mentioned also other approaches like the HTTP, I have done another test by triggering the timer function with Postman through a POST call using the master key. This test works without any issue, like when it is autotriggered.

So, at the moment, I’ve only been able to reproduce this problem when triggering the function manually from the Azure Portal.

+1
ichesa on Feb 4, 2021

@davidmrdavid

Does this mean that it worked for you with 3.0.15185 but not for 3.0.14916 or vice-versa?

It did not work with the version 3.0.15185, but it worked when I lowered the version to 3.0.14916. I’m sorry it’s not clear.

+1
shibayan on Jan 14, 2021
Read more comments on GitHub
← azure-functions-durable-extension: Function App instance with a single Durable Entity in it produced tons of 'EventRaised' records in its *History table
azure-functions-durable-extension: Functions always fails with InvalidCastException when distributed tracing is enabled →