azure-sdk-for-net: ServiceBusTrigger continously logs warnings about "The operation did not complete within the allocated time 00:01:00 for object receiverNNN"
We have a Function App running in Azure, with a ServiceBusTrigger configured. And while it is running, twice every minute, it will log a warning for:
RunOperation encountered an exception and will retry. Exception: Azure.Messaging.ServiceBus.ServiceBusException: The operation did not complete within the allocated time 00:01:00 for object receiver596. (ServiceTimeout)
---> System.TimeoutException: The operation did not complete within the allocated time 00:01:00 for object receiver596.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
at Microsoft.Azure.Amqp.AmqpObject.<>c.<OpenAsync>b__51_1(IAsyncResult r)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenAmqpObjectAsync(AmqpObject target, TimeSpan timeout, CancellationToken cancellationToken, String entityPath)
at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenReceiverLinkAsync(String identifier, String entityPath, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken)
at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.OpenReceiverLinkAsync(TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String identifier, CancellationToken cancellationToken)
at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.<>c.<<OpenLinkAsync>b__72_0>d.MoveNext()
--- End of stack trace from previous location ---
at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
Category: Azure.Messaging.ServiceBus LogLevel: Warning EventName: RunOperationExceptionEncountered
I spent about 1 working-day troubleshooting this, until I realized it is actually fully functional, and I guess the behavior is part of the intented implementation. BUT logging this with loglevel warning should not be done then.
Nuget packages: Microsoft.Azure.WebJobs.Extensions.ServiceBus, 5.2.0 Microsoft.NET.Sdk.Functions, 4.0.1
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 21 (7 by maintainers)
I would agree that for this specific retriable error, logging as verbose would be sufficient. In general, I think retries for exceptions that are not expected during the normal application flow should be Informational. I will take a look and see what we can do to improve this.
@JoshLove-msft I totally agree that these messages should be at the Debug level. I don’t want to have to filter out this namespace specifically at the Informational level as there may be other messages that I do want to log. As no action can be performed by the developer to change whether or not these messages would appear, I see them as diagnostic and part of normal correct operation rather than something which should be logged with such frequency.
@JoshLove-msft, Now I’m getting into details here, but I put it out there and you can ignore it if you don’t agree:
I usually like to have my log settings to write all informational, and more serious, to my logs. But I have seen that with the Functions SDK/Framework/Host a lot of “noise” is then written to the log. Things that I myself would have classified as “Trace” or “Debug” level log messages. In this case: something happens, it is by design and fully intentional, AND happens every minute - should that be considered of “Informational” log level? I don’t think so - I think it is just an implementation detail and should be either Trace or Debug log level.
Since there are 2 of these messages for every BlobTrigger function, every minute, and we have ~10 of these - that is 864 000 of these log messages per month!
Your own definition:
from https://docs.microsoft.com/en-us/azure/azure-functions/configure-monitoring?msclkid=46fd565fc06c11ecaf69608b1789989d&tabs=v2#configure-log-levels
Has it really any “long-term value” that the app behaves exactly as intended?
I am developing an azure function running in-process based on .net6. Function is triggered by a service bus.
I can confirm that I am struggling with the same problem described by @marnilss.
When the function runs locally and directly in Azure, it logs the following warnings:
Context
Function configuration:
Nuget packages:
Seems related: https://github.com/Azure/azure-functions-servicebus-extension/issues/202 I moved from Microsoft.Azure.Functions.Worker.Extensions.ServiceBus beta 5 back to 4.2.1 and the warnings go away. Functionality is still correct.