ApplicationInsights-dotnet: TelemetryClient.Flush deadlocks

If you are reporting bug/issue, please provide detailed Repro instructions.

Repro Steps

netcoreapp2.2 console app
telemetry configuration:

ServerTelemetryChannel
DependencyTrackingTelemetryModule and UnobservedExceptionTelemetryModule

call TelemetryClient.Flush on workload completion (async Task<int> Main)

Actual Behavior

The app deadlocks during Flush operation. Occurrence is reliably about 1 in 8000 runs in a consistent hardware and deployment environment.

Expected Behavior

not deadlock

Version Info

SDK Version : appinsights 2.10.0 .NET Version : netcore 2.2 How Application was onboarded with SDK(VisualStudio/StatusMonitor/Azure Extension) : OS : win10-x64 Hosting Info (IIS/Azure WebApps/ etc) : console

Two stacks seem relevant for this: Thread1

ntdll.dll!NtWaitForMultipleObjects()	Unknown
KERNELBASE.dll!WaitForMultipleObjectsEx()	Unknown
[Managed to Native Transition]	
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.Extensibility.MetricSeriesAggregatorBase<double>.UpdateAggregate(Microsoft.ApplicationInsights.Metrics.Extensibility.MetricValuesBufferBase<double> buffer)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.Extensibility.MetricSeriesAggregatorBase<double>.CompleteAggregation(System.DateTimeOffset periodEnd)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.GetNonpersistentAggregations(System.DateTimeOffset tactTimestamp, Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.AggregatorCollection aggregators)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.CycleAggregators(ref Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.AggregatorCollection aggregators, System.DateTimeOffset tactTimestamp, Microsoft.ApplicationInsights.Metrics.Extensibility.IMetricSeriesFilter futureFilter, bool stopAggregators)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.StartOrCycleAggregators(Microsoft.ApplicationInsights.Metrics.Extensibility.MetricAggregationCycleKind aggregationCycleKind, System.DateTimeOffset tactTimestamp, Microsoft.ApplicationInsights.Metrics.Extensibility.IMetricSeriesFilter futureFilter)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricManager.Flush(bool flushDownstreamPipeline)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.TelemetryClient.Flush()	Unknown

Thread 2

ntdll.dll!NtDelayExecution()	Unknown
KERNELBASE.dll!SleepEx()	Unknown
[Managed to Native Transition]	
System.Private.CoreLib.dll!System.Threading.SpinWait.SpinOnce(int sleep1Threshold) Line 169	C#
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.Extensibility.MetricValuesBufferBase<double>.GetAndResetValue(int index)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MeasurementAggregator.UpdateAggregate_Stage1(Microsoft.ApplicationInsights.Metrics.Extensibility.MetricValuesBufferBase<double> buffer, int minFlushIndex, int maxFlushIndex)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.Extensibility.MetricSeriesAggregatorBase<double>.UpdateAggregate(Microsoft.ApplicationInsights.Metrics.Extensibility.MetricValuesBufferBase<double> buffer)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.Extensibility.MetricSeriesAggregatorBase<double>.CompleteAggregation(System.DateTimeOffset periodEnd)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.GetNonpersistentAggregations(System.DateTimeOffset tactTimestamp, Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.AggregatorCollection aggregators)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.CycleAggregators(ref Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.AggregatorCollection aggregators, System.DateTimeOffset tactTimestamp, Microsoft.ApplicationInsights.Metrics.Extensibility.IMetricSeriesFilter futureFilter, bool stopAggregators)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.MetricAggregationManager.StartOrCycleAggregators(Microsoft.ApplicationInsights.Metrics.Extensibility.MetricAggregationCycleKind aggregationCycleKind, System.DateTimeOffset tactTimestamp, Microsoft.ApplicationInsights.Metrics.Extensibility.IMetricSeriesFilter futureFilter)	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.DefaultAggregationPeriodCycle.FetchAndTrackMetrics()	Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Metrics.DefaultAggregationPeriodCycle.Run()	Unknown
System.Threading.Thread.dll!System.Threading.Thread.ThreadMain_ThreadStart() Line 93	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 167	C#
[Native to Managed Transition]	
kernel32.dll!BaseThreadInitThunk()	Unknown
ntdll.dll!RtlUserThreadStart()	Unknown

About this issue

Original URL
State: open
Created 5 years ago
Comments: 29 (14 by maintainers)

Most upvoted comments

Customer request to add this item to 2.21 release if possible.

toddfoust on Mar 16, 2022

We have the latest version now running for 5 months, and the issue did not occur anymore. Thanks for looking into it.

BertscheS on Jan 11, 2023

Apologies for the delay. We have this scheduled for the upcoming sprint, since we saw the deadlock again in integration. So we will upgrade to the new library.

From: Cijo Thomas @.> Sent: Friday, September 16, 2022 2:44 PM To: microsoft/ApplicationInsights-dotnet @.> Cc: [ɴιɴlαr] @.>; Comment @.> Subject: Re: [microsoft/ApplicationInsights-dotnet] TelemetryClient.Flush deadlocks (#1186)

If anyone who experienced this issue can confirm that the latest version fixes the issue, it would be greatly appreciated.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/ApplicationInsights-dotnet/issues/1186#issuecomment-1249852836, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAIHCW3I6CHO43Q6NYIZQP3V6TS5RANCNFSM4IISHSYA. You are receiving this because you commented.Message ID: @.***>

ninlar on Sep 16, 2022

We have created a backlog item to upgrade the SDK and test the fix.

ninlar on Jul 26, 2022

This is consistently happening to us in production also. We are calling flush from two places and in some scenarios that seems to cause the deadlock (it may be actually livelock). I am attaching stack traces from debugger, they start at the bottom after the calls from our code to the sdk. When I try to resume and pause again the debugger, the code doesn’t seem to move (that however doesn’t tell us much since it just may be improbable for me to hit anything but the spinwait).

This manifests when the application should shut down as both calls to flush are started only at that point. In most cases this doesn’t happen and shutdown completes normally. When the deadlock occurs, metrics about cpu and memory usage are still sent.

We are using only services.AddApplicationInsightsTelemetryWorkerService() with no default overrides.

(sorry for these being screens but vscode ignores me when I ask it to “copy the stack trace”)

Similar this is happening for my code as well. Have used the below settings

var aiOptions = new Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions { InstrumentationKey = instrumentationKey, EnableAdaptiveSampling = false, EnableQuickPulseMetricStream = false, EnablePerformanceCounterCollectionModule = false, };

but still getting high number of threads and results in 100% cpu. See below for thread details.

any updates would be helpful. In the meantime , will try ‘AddAutoCollectedMetricExtractor’ to false and monitory.

shoguns6 on Feb 22, 2021