keda: Azure Storage Queue scaler is reporting wrong number of messages in queue

Report

Hello I think the azure_queue monitor is not reading the number of messages in the queue as it should Looking at the code here I think that when Keda calculates the “visible messages” as 0 it returns 0, even though the actual number of messages in the queue is much higher

For example: We see “showing 0 of 128503 messages in queue” in the portal, and Keda is also reporting 0 for the metric (even though the actual count is 128k) - see the right side in the screenshot When we work with the c# SDK for azure storage we’re getting the right numbers - see left side in the screen shot (csharp code below):

image

public async Task Run()
        {
            m_tracer.TraceInformation($"{nameof(StorageAccountMetricsLogger)} - Starting work.");
            CloudStorageAccount storageAccount = m_aadStorageProvider.GetStorageAccount(m_queueStorageName);
            var queueClient = storageAccount.CreateCloudQueueClient();
            var queueNames = queueClient.ListQueues();
            foreach (var cloudQueue in queueNames)
            {
                var dimension = new Dimensions();
                dimension.SetDimension("Name", cloudQueue.Name);
                var instrumentor = new Instrumentor(dimension);
                await cloudQueue.FetchAttributesAsync();
                if (!cloudQueue.ApproximateMessageCount.HasValue)
                {
                    m_tracer.TraceWarning(0, $"{nameof(StorageAccountMetricsLogger)} - Failed to retrieve message count for storage queue {cloudQueue.Name}.");
                    continue;
                }
                instrumentor.LogMetric(s_monitoringStorageQueueActiveMessageCountMetricName, cloudQueue.ApproximateMessageCount.Value);
                m_tracer.TraceInformation($"{nameof(StorageAccountMetricsLogger)} - Metrics for storage queue name \"{cloudQueue.Name}\" messages = {cloudQueue.ApproximateMessageCount}");
            }
            m_tracer.TraceInformation($"{nameof(StorageAccountMetricsLogger)} - Done.");
        }

Expected Behavior

keda should report the actual # of messages in queue

Actual Behavior

keda is reporting 0 messages in queue

Steps to Reproduce the Problem

  1. add messages to azure queue
  2. query the queue with keda

Logs from KEDA operator

No response

KEDA Version

2.4.0

Kubernetes Version

1.19

Platform

Other

Scaler Details

Azure storage queue

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Did anything happen with this?

I am in exactly the same situation as jungopro both with the scaling issue with visible messages, and the viability of scaled jobs.

Not including invisible messages doesn’t seem right. If I have KEDA set to scale 5 messages per pod, and I have 5 messages in progress, it would take another 6 messages to make KEDA request a second pod. It also results in the race condition observed here were if all of the in progress messages are not completed within the cooldown time frame, KEDA will scale in while they are still processing.

Having the option to use approximate message count instead seems ideal.

On a side note: when using Azure functions on an App Service Plan, you can scale based on storage queue length. The implementation Azure has used for this is approximate message count, and does not offer any other method of scaling based on queue length.

A screenshot from the app service plan scale rules GUI: image

hey @antonio-policastro This is the .NET docs about that method.

Right. Approximate may return >0 count when in fact there aren’t any messages at all. This looks more and more like something that we can resolve with a scaler parameter that will determine whether to use peek or not

On Tue, Mar 1, 2022 at 2:34 PM Jorge Turrado Ferrero < @.***> wrote:

I’m not sure about it, peek returns the value with accuracy. ApproximateMessagesCount (at least in .NET SDK) could return more messages than actual count

— Reply to this email directly, view it on GitHub https://github.com/kedacore/keda/issues/2385#issuecomment-1055398053, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATJ7GSL2JAHS5X5HKN2V7LU5YFGJANCNFSM5JMYOAVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>