azure-functions-host: ITelemetryProcessor does not appear to be supported in Function Apps

Please factor the following while weighing the pros and cons of tracking successful dependencies:

These are screenshots of the Application Insights instance and cost analysis for a particular Function App and related storage account in our system.

Above is a capture of the default 24-hour period search for the service in question. You can see that dependency tracking accounts for 1.4 million items, while Trace, Request, and Exception account for 40K, 18K, and 1.9K (really must look in to those exceptions), respectively. Dependency events account for approximately 96% of all events.

This is the cost projection of the Application Insights instance. As before, the image shows that “REMOTEDEPENDENCY” events make up for the vast majority of recorded events.

Finally, the above screenshot is a filtered selection from the “Cost by Resource” view, showing the cost of the Function App, Storage Accounts, and Application Insights instances in question. The cost of the Application Insights instance is 1252% that of the cost of the Function App it is monitoring.

These costs are woefully unsustainable for us. Of late, my decision to use Function Apps, which I’d touted to my colleagues as extremely cost effective, is being called in to question by my teammates and superiors. Application Insights has been an invaluable tool for diagnostics and troubleshooting. I’d liken using a Function App without an associated Application Insights instance to flying by the seat of my pants. That said, I will eventually have to choose to either stop using Application Insights, or stop using Function Apps. I’m sure I’m not the only one who would really appreciate if the Functions and Application Insights teams could find a solution by which that choice doesn’t have to me made.

I have a related problem so I thought I’d write about it here rather than make a new issue. In my function app, there are many calls to dependent services. These calls are all tracked as Dependencies in Application Insights. The vast majority of data in my Application Insights instances are from successful dependency calls. The cost of tracking these calls has become substantial.

I’ve tried implementing the ITelemetryProcess interface to filter them out, and then later found here that this doesn’t work.

Is there some other way to disable Dependency Tracking in Application Insights for Function Apps? Currently, my two options are 1) Pay a substantial amount of money to needlessly track dependency successes, or 2) get rid of Application Insights.

We are also impacted by this issue and have tried several options to fix this behaviour. Our Functions are sitting behind FrontDoor and we are using the built-in Health check option to probe our services. Since Azure has a significant amount of Point of Presence around the world (which is great), this results in almost 200k logs per day per backend pool on the lowest setting (probing at a 255 second interval).

Using the built-in filtering option, we were only able to hide the traces (Executing ‘HealthProbe’ / Executed ‘Health Probe’) but couldn’t find any out of the box way to filter out the ~70k requests generated in the process.

@gabrielweyer solution is indeed allowing us to filter out all the unwanted requests but forces us to import a new package and fix our custom implementation or bring in a lot of code that may break with future releases.

Before we spend more time looking into this and implementing an alternative, is there any update on this issue or future plans to allow developers to filter out requests by name? Is this active PR still being looked at? https://github.com/Azure/azure-webjobs-sdk/pull/2657

I am also having an issue getting this working in Azure.

I have implemented a small Azure Function using a version of the code outlined by @michaeldaw (thank you!) and can see that it is working when I execute locally, but when it is deployed to Azure it has no effect and all the dependency messages are appearing in AI.

Configuration

• .Net Core 2.2
• Custom Directory.Build.targets to ensure the correct extensions.json gets deployed (see https://github.com/Azure/azure-functions-host/issues/3386)

Packages

<PackageReference Include="Microsoft.ApplicationInsights" Version="2.7.2" />
<PackageReference Include="Microsoft.ApplicationInsights.PerfCounterCollector" Version="2.7.2" />
<PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.1.1" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.28" />
<PackageReference Include="MimeTypesMap" Version="1.0.7" />
<PackageReference Include="System.Data.SqlClient" Version="4.4.0" />

The reason I want to do this is that we make calls to Azure Blob storage to add/remove files including a call to await blob.CheckIfExistsAsync() which raises a 404 dependency error filling up our logs with issues that are not real (and wasting my time trying to figure out where these “errors” were coming from.

I have tried using newer and older versions of the Microsoft.ApplicationInsights libraries, using .Net Core 2.1 and .Net Standard 2.0, nothing effects the AI logging and I am not sure why.

As a side note, I have a related issue where the logging level in the host.json is ignored by AI as well (raised as a separate query here: https://github.com/Azure/azure-functions-host/issues/4474). I include it here as it may indicate some configuration problem that may be causing this issue.

I came up with a solution that calls telemetry processors on request telemetry items while re-using the existing processor chain (so that the Functions internal dependencies are discarded before hitting the user’s telemetry processors). I was hoping that Azure Functions v4 would support telemetry processors but it does not (at least for the in-process hosting model). v4 emits a bit more telemetry and Azure.Core emits a lot more telemetry.

I understand that using telemetry processors can skew the statistics and that sampling is the preferred approach but sometimes I need to discard a specific telemetry item and I’m happy for sampling to apply to everything else. Without any configuration some Functions will incur an Application Insights bill 10 times bigger than the compute bill!

Adding telemetry processors support to Azure Functions would make it a more compelling offering. Currently we have to take into consideration the additional Application Insights spending.

The solution I came up with has been inspired by the other workarounds posted on this thread. I created a library out of it which hopefully will be easy enough to integrate into any Functions App. The code is a bit too long to post here, the interesting part is the extension method configuring Application Insights.

Using this customisation:

• You can add any numbers of telemetry processors in the same way you would add them in ASP.NET Core
• When APPINSIGHTS_INSTRUMENTATIONKEY / APPLICATIONINSIGHTS_CONNECTION_STRING is not present, I register a no-op TelemetryConfiguration so that it can be resolved by the container (no more exception when Application Insights is not configured)
• Most of the “cruft” telemetry emitted by the runtime is discarded

The project comes with a couple of sample Functions so that you can see how it behaves.

Hopefully this helps someone else, it has helped us reduce our Application Insights spending while retaining the ability to troubleshoot issues.

We struggle with very high cost for application insights and the solutions and workaround proposed for this issue would help us to reduce the cost and still be able to use application insights. In my opinion, filtering should be possible with configuration, without need to add code which has a high chance to break with future SDK or Function runtime releases…

The fact that this issue is open for almost four and a half year now will have some weight in a decision for another monitoring solution…

Additionally, I would expect to learn all this things I can read in this thread in the documentation without having to read through a GitHub-issue!

@luthus’s solution above is the correct one, but if you don’t want to have to fork the web jobs sdk, you can get it working correctly in durable functions (or azure functions in general) WITHOUT breaking live metrics dependency and error logging like @kamilzzz saw and I did as well with @lmolkova’s solution.

Adding the ITelemetryModule and inserting your ITelemetryProcessors into the builder there works as expected. This gets around builder.Services.AddApplicationInsightsTelemetryProcessor() not working as expected.

// startup.cs
builder.Services.AddSingleton<ITelemetryModule, MyCustomTelemetryModule>();
builder.Services.AddApplicationInsightsTelemetry(Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY"));

// custom module
public class MyCustomTelemetryModule : ITelemetryModule
{
	public void Initialize(TelemetryConfiguration configuration)
	{
		// add custom processors
		configuration.TelemetryProcessorChainBuilder.Use(next => new MyCustomTelemetryProcessor(next));
		configuration.TelemetryProcessorChainBuilder.Build();
	}
}

// custom processor
public class MyCustomTelemetryProcessor : ITelemetryProcessor
{
	private readonly ITelemetryProcessor _next;

	public MyCustomTelemetryProcessor(ITelemetryProcessor next)
	{
		_next = next;
	}

	public void Process(ITelemetry item)
	{
                bool myCustomSkipTelemetry = false;
		if (myCustomSkipTelemetry)
			return;
		_next.Process(item);
	}
}

@michaeldaw hit the nail on the head. We’re having the exact same issue and would love to be able to reduce our AI bill by filtering out records. Particularly filtering out successful dependency calls would be a big positive.

For now we’re going to have to replace our logging implementation to write useful data to another logging service until this gets some traction.

This change appears to resolve the issue: https://github.com/luthus/azure-webjobs-sdk/commit/3137c5c8e59fdd2495c4ad0b4a09b8748f7ee1f9

With this change we should be able to use builder.Services.AddApplicationInsightsTelemetryProcessor<T>() in the same way that we do for ASP.NET Core.

Everything under "applicationInsights" maps directly to the ApplicationInsightsLoggerOptions class. "enableDependencyTracking" needs to go under "applicationInsights", not under "samplingSettings".

I’ve submitted a PR to the docs: https://github.com/MicrosoftDocs/azure-docs/pull/43025.

Hi @Areson, I was able to filter out dependencies using a cutom ITelemetryInitializer in the code above (based on the builder.Services.AddSingleton<ITelemetryInitializer, CustomTelemetryInitializer>(); line that you have), I implemented a CustomTelemetryInitializer like this:

public class CustomTelemetryInitializer : ITelemetryInitializer
{
  /// <summary>
  /// Ignore the following as they are noise and skew the telemetry data:
  /// - 404 errors from dependencies (such as Azure Blob storage) as these are triggered when checking if items exists
  /// Checks if the value passed is a <see cref="DependencyTelemetry"/> and the <see cref="DependencyTelemetry.ResultCode"/> = 404
  /// </summary>
  /// <param name="telemetry">The ITelemetry value to test</param>
  public void Initialize(ITelemetry telemetry)
  {
    if (telemetry is DependencyTelemetry dependency && dependency != null && dependency.ResultCode.Equals("404"))
    {
      dependency.Success = true;
    }
  }
}

That coupled with the custom ITelemetryProcessor to filter specific trace messages enabled me to capture just the specific info I wanted.

Hope this helps 😃

P.S. See my previous post on replacing TelemetryConfiguration.Active which is deprecated in the latest version of Microsoft.Azure.WebJobs.Logging.ApplicationInsights

I have got it working again with the latest Microsoft.Azure.WebJobs.Logging.ApplicationInsights (version 3.0.14) without neededing to call the now obsolete TelemetryConfiguration.Active

For the benefit of others, here is the sample working code:

public class Startup : IWebJobsStartup
{
    public void Configure(IWebJobsBuilder builder)
    {
        var configDescriptor = builder.Services.SingleOrDefault(tc => tc.ServiceType == typeof(TelemetryConfiguration));
        if (configDescriptor?.ImplementationFactory != null)
        {
            var implFactory = configDescriptor.ImplementationFactory;
            builder.Services.AddSingleton<ITelemetryInitializer, CustomTelemetryInitializer>();
            builder.Services.Remove(configDescriptor);
            builder.Services.AddSingleton(provider =>
            {
                if (implFactory.Invoke(provider) is TelemetryConfiguration config)
                {
                    // Construct a new TelemetryConfiguration based off the existing config
                    var newConfig = new TelemetryConfiguration(config.InstrumentationKey, config.TelemetryChannel)
                    {
                        ApplicationIdProvider = config.ApplicationIdProvider
                    };
                    // Add the telemetry initializers (including any custom ones added above)
                    config.TelemetryInitializers.ToList().ForEach(initializer => newConfig.TelemetryInitializers.Add(initializer));
                    // Add custom telemetry processor first
                    newConfig.TelemetryProcessorChainBuilder.Use(next => new CustomTelemetryProcessor(next));
                    // Add existing telemetry processors next (excluding any default processors that are already registered)
                    foreach (var processor in config.TelemetryProcessors.Where(processor => !newConfig.TelemetryProcessors.Any(x => x.GetType() == processor.GetType())))
                    {
                        newConfig.TelemetryProcessorChainBuilder.Use(next => processor);
                        if (processor is QuickPulseTelemetryProcessor)
                        {
                            // Re-register the QuickPulseTelemetryProcessor (reguired as we will re-build the chain below)
                            var quickPulseModule = new QuickPulseTelemetryModule();
                            quickPulseModule.RegisterTelemetryProcessor(processor);
                        }
                    }
                    // Build the chain after adding the telemetry processors 
                    newConfig.TelemetryProcessorChainBuilder.Build();
                    newConfig.TelemetryProcessors.OfType<ITelemetryModule>().ToList().ForEach(module => module.Initialize(newConfig));
                    return newConfig;
                }
                return null;
            });
        }
    }
}

Key changes are:

1. TelemetryConfiguration.Active replaced with new TelemetryConfiguration(config.InstrumentationKey, config.TelemetryChannel)
2. Setting of TelemetryInitializers now required as it is not pulled across from Active

Hope this helps someone and it would be useful if the MS docs were updated to reflect this (there are still a lot recommending the use of TelemetryConfiguration.Active, especially with regard to Azure functions).

@dzhang-quest Thank you very much for posting that code. It’s working great. That was really the final piece of the puzzle. I’d thought I’d post the complete class in case anyone is interested. The programmer should be able to just add this class to their Azure Function project.

using MyNamespace;
using System.Linq;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse;

[assembly: WebJobsStartup(typeof(Startup))]

namespace MyNamespace
{
    public class Startup : IWebJobsStartup
    {
        public void Configure(IWebJobsBuilder builder)
        {
            var configDescriptor = builder.Services.SingleOrDefault(tc => tc.ServiceType == typeof(TelemetryConfiguration));
            if (configDescriptor?.ImplementationFactory == null)
                return;

            var implFactory = configDescriptor.ImplementationFactory;

            builder.Services.Remove(configDescriptor);
            builder.Services.AddSingleton(provider =>
            {
                if (!(implFactory.Invoke(provider) is TelemetryConfiguration config))
                    return null;

                config.TelemetryProcessorChainBuilder.Use(next => new AiDependencyFilter(next));
                config.TelemetryProcessorChainBuilder.Build();

                var quickPulseProcessor = config.TelemetryProcessors
                    .OfType<QuickPulseTelemetryProcessor>()
                    .FirstOrDefault();

                if (!(quickPulseProcessor is null))
                {
                    var quickPulseModule = provider
                        .GetServices<ITelemetryModule>()
                        .OfType<QuickPulseTelemetryModule>()
                        .FirstOrDefault();

                    quickPulseModule?.RegisterTelemetryProcessor(quickPulseProcessor);
                }

                config.TelemetryProcessors
                    .OfType<ITelemetryModule>()
                    .ToList()
                    .ForEach(module => module.Initialize(config));

                return config;
            });
        }
    }

    internal class AiDependencyFilter : ITelemetryProcessor
    {
        private readonly ITelemetryProcessor _next;
        public AiDependencyFilter(ITelemetryProcessor next)
        {
            _next = next;
        }

        public void Process(ITelemetry item)
        {
            var request = item as DependencyTelemetry;

            if (request?.Name != null)
            {
                return; 
            }

            _next.Process(item);
        }
    }
}

So I have to filter out an exception, a false positive (for details see https://github.com/Azure/azure-webjobs-sdk/issues/2612, the issue is completely ignored for over half a year now), filtering this exception is a workaround all together, since this exception should not be logged at all… But anyways…

I can not use category filtering because exception is thrown by Entity Framework code, the category is Microsoft.EntityFrameworkCore.Update and I want to log this exceptions except exceptions with specific message.

I developed a telemetry processor and added it using custom ITelemetryModule approach, and it seems to work locally. Unfortunately when deployed to Azure exceptions are not being filtered. I use Microsoft.ApplicationInsights.AspNetCore version 2.15 and I can not downgrade it to 2.6.1 as suggested here because I also use EnableSqlCommandTextInstrumentation configuration

@brettsam @lmolkova the issue is not fixed for over 2 years now. This issue makes us loose money on AI fees and even more money on developing and researching workarounds. Could we please have this issue fixed once and for all? Pretty please?

Would it be possible to get the PR for this (https://github.com/Azure/azure-webjobs-sdk/pull/2657) reviewed and merged so users can start utilizing custom telemetry processors? I am in the process of trying to stand up a new functions app that is registering very large numbers of dependency telemetry so utilizing a custom filter to exclude this is a must have.

We’re running into the issue described in https://github.com/Azure/azure-functions-host/issues/3741#issuecomment-584902726 issue locally (haven’t tried Azure yet) when using Functions V3. We are trying to filter out synthetic traffic (FrontDoor/Traffic Manager probes) using a custom telemetry processor. Currently, since there is no service type of “TelemetryConfiguration” being registered, we can’t hook into it and add our own processor. We’re using the 3.0.7 SDK version, and I tried using the 2.11.0 version of App Insights, but that didn’t fix it.

@lmolkova: thank you for explaining this. In our case, the functions in question are HTTP-triggered. The dependencies in questions are, for the most part, calls to tables using the .net storage API. It sounds like these kinds of calls would fall under the second part of your explanation (http calls, etc.).

I’ve modified the original AiDependencyFilter class that I posted earlier to allow failed dependency calls through the filter. I’ve only tested it a little bit, but it seems to restore the end-to-end tracing at least for those function calls. This is entirely sufficient for my purposes. It’s really only when the calls fail that I’m interested in the tracing.

Thank you for your help.

I apologize for the delay – I lost track of this issue. Looping in @lmolkova for her thoughts as well.

For the core issue filed – registering a custom ITelemetryProcessor should be do-able and fairly easy. You could register one or more with DI and we could stick it at the end of the processor chain, after all of our built-in processors run. That shouldn’t be too difficult and as long as we clearly document where it runs in the chain, I think it makes sense. It would allow a lot of what I’ve listed out below to be done in a custom way by anyone.

There’s a couple of other issues brought up that are good as well:

• @roryprimrose: looking back up at this comment, I think I see the issue. You’re using the same TelemetryClient that we create via DI, which runs a custom ITelemetryInitializer. This will grab the current logging scope and apply it to the telemetry being logged. Because you’re doing this in a background task, that likely explains why you’re getting an incorrect category. If you, instead, created your own TelemetryClient there (using the same instrumentation key – which can be pulled from an injected IConfiguration) – you should get category-less metrics that you can log however you want.
• Although not directly brought up, it may make sense to log metrics with a different category – like Function.FunctionName.Metric to allow for more granular filtering. They currently get lumped in as Function.FunctionName.User, which is workable but may not give enough switches for everyone.
• @michaeldaw / @KonnectSalon / @jamesdheadup: Can you share what the customDimentions.Category is for your dependencies that you’re seeing? I’m going through how we “tag” DependencyTelemetry right now and seeing a few things we can improve (I’ll move these to a separate issue but wanted to enumerate them here):
  • If the dependency fails, we don’t flip the LogLevel to Error, which we should. That would allow you to filter and only show failed dependencies.
  • We probably should add a blanket on/off for dependency auto-tracking.
  • There are scenarios where the category will be Host.Bindings and scenarios where the category will be Function.FunctionName. We should try to reconcile this for better filtering.
  • We should think about whether we want to have some threshold for slow dependencies as well – i.e. only log (as Warning?) ones that take longer than 1 second (configurable).

@brettsam Nope, sorry, I was right the first time. My custom metrics, written directly via TelemetryClient, are being recorded with the same category used to write trace records from the queue trigger.

I have a queue trigger AnalyticsHintsProcessor which logs with the category Function.AnalyticsHintsProcessor. I also have a timer trigger AnalyticsScavenger which logs with the category Function.AnalyticsScavenger.

These are the customMetrics that are recorded for both scenarios.

These are the trace records written by the triggers.

So I can’t use a logging config filter to wipe out the trace records for the storage triggers without also wiping out the custom metrics.

Why is TelemetryClient coupled to ILogger configuration filters?