azure-functions-dotnet-worker: Logs sometimes going missing in isolated worker
Description
I have migrated a .NET 6 in-process function app to a .NET 8 isolated app, following the migration guide at https://learn.microsoft.com/en-au/azure/azure-functions/migrate-dotnet-to-isolated-model.
Like many others, I have had issues with logging.
My current setup looks like:
var host = new HostBuilder()
.ConfigureFunctionsWebApplication()
.ConfigureServices(services => {
services.AddApplicationInsightsTelemetryWorkerService();
services.ConfigureFunctionsApplicationInsights();
// other DI
})
.ConfigureLogging(logging =>
{
// try to force Information logs for our name spaces
logging.AddFilter("Orchestration.FnApp", LogLevel.Information);
logging.AddFilter("Orchestration.Core", LogLevel.Information);
// try to force Information logs for our name spaces
logging.AddFilter<Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider>("Orchestration.FnApp", LogLevel.Information);
logging.AddFilter<Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider>("Orchestration.Core", LogLevel.Information);
})
.Build();
With the logging filters used to override ApplicationInsights dropping all logs below warning (I think only the second filter is required). We only want our code logging at Information level, so this should be ok.
The package versions are:
<PackageVersion Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker" Version="1.21.0" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.3.0" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.0" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.EventHubs" Version="6.2.0" />
<PackageVersion Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />
I now have 2 issues outstanding, one of which is a big problem, one a nice to have.
The big issue is that sometimes logs just don’t turn up in Application Insights. For example, I have a timer function which looks similar to below, where the first thing it does is log the time:
public class TimerFunction
{
private readonly ILogger<TimerFunction> _logger;
public TimerFunction(ILogger<TimerFunction> logger)
{
_logger = logger;
}
[Function("TimerFunction")]
public async Task RunTimerFunction([TimerTrigger("0 */15 * * * *")] TimerInfo myTimer)
{
var scheduleTime = RoundToNearest(DateTime.UtcNow, TimeSpan.FromMinutes(15));
_logger.LogWarning("Schedule time: {scheduleTime}", scheduleTime);
// rest of code
}
}
Sometime we see the log:
Sometimes we don’t:
There doesn’t seem to be any pattern to this. As far as I can see, this should be logging regardless of any Application Insights settings since it is at the Warning level.
This was also mentioned here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1182#issuecomment-1372630620, but didn’t seem to get followed up.
The second, nice to have issue is the missing log level or Type, seen in the screenshots above. This was mentioned here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1182#issuecomment-1332902091
With the solution here: https://github.com/Azure/azure-functions-dotnet-worker/issues/1182#issuecomment-1347216293 was to use a ITelemetryInitializer. A clear example of this would be useful!
Steps to reproduce
Run the app as described above on .NET 8 isolated.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15
I can now confirm that all of my logs are arriving. So this serves as a reminder that all
applicationInsightssettings fromhost.jsonneed to migrated over to the worker configuration.Also, I was able to add back the log type field by using the following
ITelemetryInitializer:And initializing it in
Program.csvia:There is no sampling applied within application insights:
But following the information below that, I see that adaptive sampling is applied in the SDK by default. I am now testing with that disabled via:
This particular function runs every 15min, so it will take a little while to see the effect.
However, I did find this: https://stackoverflow.com/questions/68026094/how-to-tell-if-my-missing-logs-are-due-to-adaptive-sampling-in-application-insig
Running the query:
Which suggests that I have been losing logs due to adaptive sampling and that maybe disabling it resolves the issue (need to wait a while yet).
All of this is a bit of a rabbit hole since it all just worked when running in-process. Maybe it needs to be mentioned in the migration guide?