azure-functions-dotnet-worker: Logs sometimes going missing in isolated worker

Description

I have migrated a .NET 6 in-process function app to a .NET 8 isolated app, following the migration guide at https://learn.microsoft.com/en-au/azure/azure-functions/migrate-dotnet-to-isolated-model.

Like many others, I have had issues with logging.

My current setup looks like:

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services => {
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();

        // other DI
    })
    .ConfigureLogging(logging =>
    {
        // try to force Information logs for our name spaces
        logging.AddFilter("Orchestration.FnApp", LogLevel.Information);
        logging.AddFilter("Orchestration.Core", LogLevel.Information);

        // try to force Information logs for our name spaces
        logging.AddFilter<Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider>("Orchestration.FnApp", LogLevel.Information);
        logging.AddFilter<Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider>("Orchestration.Core", LogLevel.Information);
    })
    .Build();

With the logging filters used to override ApplicationInsights dropping all logs below warning (I think only the second filter is required). We only want our code logging at Information level, so this should be ok.

The package versions are:

    <PackageVersion Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker" Version="1.21.0" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.3.0" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.0" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.EventHubs" Version="6.2.0" />
    <PackageVersion Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />

I now have 2 issues outstanding, one of which is a big problem, one a nice to have.

The big issue is that sometimes logs just don’t turn up in Application Insights. For example, I have a timer function which looks similar to below, where the first thing it does is log the time:

public class TimerFunction
{
    private readonly ILogger<TimerFunction> _logger;

    public TimerFunction(ILogger<TimerFunction> logger)
    {
        _logger = logger;
    }

    [Function("TimerFunction")]
    public async Task RunTimerFunction([TimerTrigger("0 */15 * * * *")] TimerInfo myTimer)
    {
        var scheduleTime = RoundToNearest(DateTime.UtcNow, TimeSpan.FromMinutes(15));
        _logger.LogWarning("Schedule time: {scheduleTime}", scheduleTime);

       // rest of code
    }
}

Sometime we see the log:

image

Sometimes we don’t:

image

There doesn’t seem to be any pattern to this. As far as I can see, this should be logging regardless of any Application Insights settings since it is at the Warning level.

This was also mentioned here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1182#issuecomment-1372630620, but didn’t seem to get followed up.

The second, nice to have issue is the missing log level or Type, seen in the screenshots above. This was mentioned here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1182#issuecomment-1332902091

With the solution here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1182#issuecomment-1347216293 was to use a ITelemetryInitializer. A clear example of this would be useful!

Steps to reproduce

Run the app as described above on .NET 8 isolated.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15

Most upvoted comments

I can now confirm that all of my logs are arriving. So this serves as a reminder that all applicationInsights settings from host.json need to migrated over to the worker configuration.

Also, I was able to add back the log type field by using the following ITelemetryInitializer:

using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;

namespace FnApp;

public class LogLevelTelemetryInitializer : ITelemetryInitializer
{
    private const string LogLevelKey = "LogLevel";

    public void Initialize(ITelemetry telemetry)
    {
        if (telemetry is not TraceTelemetry traceTelemetry)
        {
            return;
        }

        if (!traceTelemetry.Properties.ContainsKey(LogLevelKey))
        {
            var level = traceTelemetry.SeverityLevel switch
            {
                SeverityLevel.Verbose => "Verbose",
                SeverityLevel.Information => "Information",
                SeverityLevel.Warning => "Warning",
                SeverityLevel.Error => "Error",
                SeverityLevel.Critical => "Critical",
                _ => "Unknown",
            };

            traceTelemetry.Properties[LogLevelKey] = level;
        }
    }
}

And initializing it in Program.cs via:

services.AddSingleton<ITelemetryInitializer, LogLevelTelemetryInitializer>();

There is no sampling applied within application insights:

image

But following the information below that, I see that adaptive sampling is applied in the SDK by default. I am now testing with that disabled via:

        services.AddApplicationInsightsTelemetryWorkerService(options =>
        {
            options.EnableAdaptiveSampling = false;
        });

This particular function runs every 15min, so it will take a little while to see the effect.

However, I did find this: https://stackoverflow.com/questions/68026094/how-to-tell-if-my-missing-logs-are-due-to-adaptive-sampling-in-application-insig

Running the query:

union requests,dependencies,pageViews,browserTimings,exceptions,traces
| where timestamp > ago(1d)
| summarize RetainedPercentage = 100/avg(itemCount) by bin(timestamp, 1h), itemType
timestamp itemType RetainedPercentage
04/04/2024, 9:00:00.000 am dependency 100
04/04/2024, 9:00:00.000 am request 100
04/04/2024, 9:00:00.000 am trace 100
04/04/2024, 8:00:00.000 am dependency 54.638447971781304
04/04/2024, 8:00:00.000 am request 100
04/04/2024, 8:00:00.000 am trace 71.00042211903757
04/04/2024, 7:00:00.000 am dependency 76.91187181354697
04/04/2024, 7:00:00.000 am request 100
04/04/2024, 7:00:00.000 am trace 86.00804649083594
04/04/2024, 6:00:00.000 am dependency 68.74506708760852
04/04/2024, 6:00:00.000 am request 100
04/04/2024, 6:00:00.000 am trace 79.65249199817102

Which suggests that I have been losing logs due to adaptive sampling and that maybe disabling it resolves the issue (need to wait a while yet).

All of this is a bit of a rabbit hole since it all just worked when running in-process. Maybe it needs to be mentioned in the migration guide?