keda: Azure Storage Queue scaler is reporting wrong number of messages in queue
Report
Hello I think the azure_queue monitor is not reading the number of messages in the queue as it should Looking at the code here I think that when Keda calculates the “visible messages” as 0 it returns 0, even though the actual number of messages in the queue is much higher
For example: We see “showing 0 of 128503 messages in queue” in the portal, and Keda is also reporting 0 for the metric (even though the actual count is 128k) - see the right side in the screenshot When we work with the c# SDK for azure storage we’re getting the right numbers - see left side in the screen shot (csharp code below):

public async Task Run()
{
m_tracer.TraceInformation($"{nameof(StorageAccountMetricsLogger)} - Starting work.");
CloudStorageAccount storageAccount = m_aadStorageProvider.GetStorageAccount(m_queueStorageName);
var queueClient = storageAccount.CreateCloudQueueClient();
var queueNames = queueClient.ListQueues();
foreach (var cloudQueue in queueNames)
{
var dimension = new Dimensions();
dimension.SetDimension("Name", cloudQueue.Name);
var instrumentor = new Instrumentor(dimension);
await cloudQueue.FetchAttributesAsync();
if (!cloudQueue.ApproximateMessageCount.HasValue)
{
m_tracer.TraceWarning(0, $"{nameof(StorageAccountMetricsLogger)} - Failed to retrieve message count for storage queue {cloudQueue.Name}.");
continue;
}
instrumentor.LogMetric(s_monitoringStorageQueueActiveMessageCountMetricName, cloudQueue.ApproximateMessageCount.Value);
m_tracer.TraceInformation($"{nameof(StorageAccountMetricsLogger)} - Metrics for storage queue name \"{cloudQueue.Name}\" messages = {cloudQueue.ApproximateMessageCount}");
}
m_tracer.TraceInformation($"{nameof(StorageAccountMetricsLogger)} - Done.");
}
Expected Behavior
keda should report the actual # of messages in queue
Actual Behavior
keda is reporting 0 messages in queue
Steps to Reproduce the Problem
- add messages to azure queue
- query the queue with keda
Logs from KEDA operator
No response
KEDA Version
2.4.0
Kubernetes Version
1.19
Platform
Other
Scaler Details
Azure storage queue
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 15 (7 by maintainers)
Commits related to this issue
- Azure queue: ApproximateMessagesCount instead of Peek (#2385) — committed to antonio-policastro/keda by tonypolik 2 years ago
- Azure queue: ApproximateMessagesCount instead of Peek (#2385) — committed to antonio-policastro/keda by antonio-policastro a year ago
Did anything happen with this?
I am in exactly the same situation as jungopro both with the scaling issue with visible messages, and the viability of scaled jobs.
Not including invisible messages doesn’t seem right. If I have KEDA set to scale 5 messages per pod, and I have 5 messages in progress, it would take another 6 messages to make KEDA request a second pod. It also results in the race condition observed here were if all of the in progress messages are not completed within the cooldown time frame, KEDA will scale in while they are still processing.
Having the option to use approximate message count instead seems ideal.
On a side note: when using Azure functions on an App Service Plan, you can scale based on storage queue length. The implementation Azure has used for this is approximate message count, and does not offer any other method of scaling based on queue length.
A screenshot from the app service plan scale rules GUI:
hey @antonio-policastro This is the .NET docs about that method.
Right. Approximate may return >0 count when in fact there aren’t any messages at all. This looks more and more like something that we can resolve with a scaler parameter that will determine whether to use peek or not
On Tue, Mar 1, 2022 at 2:34 PM Jorge Turrado Ferrero < @.***> wrote: