azure-functions-dotnet-worker: Worker shutdown may lead to failed function invocations with `ObjectDisposedException` (which may be persisted by the WebJobs listener)
Description
When running the attached reproduction for a while (usually after a few hours) in functions using Linux (we tried it in Switzerland North) the following exception occurs
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ServiceBusMessageHandler_workday_validation_queue
---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48
at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack: at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in D:\a\_work\1\s\src\DotNetWorker.Core\Context\DefaultFunctionContext.cs:line 48
at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Hosting\WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 101
at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:line 20
at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 53
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 527
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
--- End of inner exception stack trace ---
at Microsoft.Azure.WebJobs.ServiceBus.SessionMessageProcessor.CompleteProcessingMessageAsync(ServiceBusSessionMessageActions actions, ServiceBusReceivedMessage message, FunctionResult result, CancellationToken cancellationToken)
at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.ProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
at Azure.Messaging.ServiceBus.ServiceBusProcessor.OnProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
at Azure.Messaging.ServiceBus.ServiceBusSessionProcessor.OnProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
at Azure.Messaging.ServiceBus.SessionReceiverManager.OnMessageHandler(EventArgs args)
at Azure.Messaging.ServiceBus.ReceiverManager.ProcessOneMessage(ServiceBusReceivedMessage triggerMessage, CancellationToken cancellationToken)
At the moment we don’t think it is anything particular to do with the session in ASB or the ASB integration but rather with the function context handling in the middleware.
It could also be that it is related to the other issues in regard to ObjectDisposedExceptions.
https://github.com/Azure/azure-functions-dotnet-worker/issues/1929
I have already raised two PRs surrounding my research in this area but I’m unsure if they’ll help
https://github.com/Azure/azure-functions-dotnet-worker/pull/2686 https://github.com/Azure/azure-functions-dotnet-worker/pull/2685
I have also made a comment about the use of TaskCompletionSource in the synchronization logic between the middleware.
Steps to reproduce
Run the repro on Service Bus for a while Experiment.zip
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 28 (8 by maintainers)
We are also experiencing the same issue. Our function app is running on a Linux-hosted consumption plan in the West Europe region.
For reference:
@jviau On the contrary: It’s not just background noise. The impact is, that messages are not being processed (and in our case therefore deadlettered). The impact is quite high! I urge you to reconsider your priorisation on this issue.
It makes having a session queue handler running on .net 8 in isolation mode unusable.
UPDATE: We stopped having this error on September 21st 2024 from 18:00 EST. We didn’t change anything, we even reverted back the original code afterwards, and we haven’t had any problem since.
It happened on ~10 differents Function Apps, and it all stopped at once. While I’m glad this is no longer an issue, I’m pretty sure this is something happening in the “other” layer of the Function App Service. Hardware or routing maybe?
We’re getting the same kind of errors all over the place when using consumption plans; no behaviour so far that could help identify the faulting process. It fails about 5% of the time, randomly on our Azure Function App in consumption plan. Using App Service Plan on those function resolve the issue, but we would like to keep on using consumption plans.
Issue started a few weeks ago when updating references to latest versions.
Sample exception: