azure-functions-dotnet-worker: Multiple Issues of FunctionTimeOut and ServiceBusMessages not Consuming in dotnet isolated ServiceBusTriggered function -- Urgent
Hi,
I have a functionapp which has service bus triggered function.
ServiceBus Triggered function sometimes kind of get stuck and doesn’t consume messages from session based queue and when I check in appinsights i cant see any logs but can see functiontimeout exception and Language worker process restart or Did not find any initialized language workers.
It doesn’t run and over appinsights i cant even see the complete logs why it got timedout as it seems it never actually triggered my code. Because, in my code, the first line is of logging a trace saying- Message received and also logs the message and when it completes or throw any exception it will log the trace again that it completed or threw any exception. But it never logs these traces. It just logs Severity level: Information, Message: Executing ‘Functions.MyServiceTriggered’ that’s it nothing else and after 30 minutes as i run on Dedicated plan so default time out is of 30 minutes.
Please help what should I do, it seems every day i have to come online as some messages get stuck in active queue and not getting consumed so I have either restart whole functionApp or disable/enable the function to consume the messages.
Exception Examples-
`Full Exception : Microsoft.Azure.WebJobs.Host.FunctionTimeoutException /Timeout value of 00:30:00 was exceeded by function /Functions.MyFunction
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryHandleTimeoutAsync(Task invokeTask,CancellationToken shutdownToken,Boolean throwOnTimeout,CancellationToken timeoutToken,TimeSpan timeoutInterval,IFunctionInstance instance,Action onTimeout) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /663
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /571
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /527
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /306
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /352
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /108`
Another Exception `Full Exception : Exception while executing function /Functions.MyFunction —> System.InvalidOperationException /Did not find any initialized language workers
Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcherLoadBalancer.GetLanguageWorkerChannel(IEnumerable`1 rpcWorkerChannels) /_/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcherLoadBalancer.cs /27
async Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher.InvokeAsync(ScriptInvocationContext invocationContext) /_/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs /400
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs /97
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs /82
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs /20
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /581
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /527
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs /306 End of inner exception
Microsoft.Azure.WebJobs.ServiceBus.SessionMessageProcessor.CompleteProcessingMessageAsync(ServiceBusSessionMessageActions actions,ServiceBusReceivedMessage message,FunctionResult result,CancellationToken cancellationToken)
async Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.ProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Azure.Messaging.ServiceBus.ServiceBusProcessor.OnProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Azure.Messaging.ServiceBus.ServiceBusSessionProcessor.OnProcessSessionMessageAsync(ProcessSessionMessageEventArgs args)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Azure.Messaging.ServiceBus.SessionReceiverManager.OnMessageHandler(EventArgs args)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Azure.Messaging.ServiceBus.ReceiverManager.ProcessOneMessage(ServiceBusReceivedMessage triggerMessage,CancellationToken cancellationToken)`
Packages added in Project
<PackageReference Include="Azure.Messaging.EventGrid" Version="4.17.0" /> <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.21.0" /> <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.21.0" /> <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.0.0" /> <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.19.0" /> <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.11.0" /> <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.15.1" /> <PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
We are using .net 7 isolated function. We are on Dedicated AppService Plan with Always On. Please find the link for the minimum repo- Link for Repo
Host.json is
{ "version": "2.0", "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": true, "excludedTypes": "Request" }, "enableLiveMetricsFilters": true } }, "extensions": { "serviceBus": { "sessionIdleTimeout": "00:00:01", "maxAutoLockRenewalDuration": "01:05:00" } }, "functionTimeout": "01:05:00" }
Please help me with what could be the issue. We are facing this repeatedly.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 44 (14 by maintainers)
We also ran into the exact same issue. We created a new dotnet 8 isolated function with ServiceBusTriggers.
The ServiceBusTriggers stops triggering on new message when the function is idle for 15 to 30 mins. After downgrading Microsoft.Azure.Functions.Worker.Sdk to version 1.15.1 (as mentioned above), the ServiceBusTriggers keeps on working as expected
@jaliyaudagedara updated as requested!
Closing this issue as intended by the automation. Issue 2248 is unrelated to this. Here, all questions appear to have been answered, and there doesn’t seem to be any work to track here at this point.
Yes, please follow the recommendations from support team who is investigating your case.
Hi @kshyju , Whenever the function timeout comes, if you see the logs at that time, it wont even trigger our source code… Because not a single log says that source code is running. It just gets stuck at Executing and then gets timedout after 30 minutes which is default. My source code doesnt run whenever these timeout occurs.
Please check and help.