keda: FailedGetExternalMetric unable to fetch metrics from external metrics API: no matching metrics found (Azure Log Analytics)

Report

I have a scaled object with Azure Log Analytics. The auto-scaling is working and our application can scale based on Azure Log Analytics query result. But, I see a persistent warning inside the events of the HPA :

Warning FailedGetExternalMetric 3m47s (x333 over 4h3m) horizontal-pod-autoscaler unable to get external metric <namespace>/s0-azure-log-analytics-<id>/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kedaloganalytics-consumer-scaled-object-dev,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API : no matching metrics found for s0-azure-log-analytics-<id>

Describe of the hpa :

Name:                                                                                    keda-hpa-kedaloganalytics-consumer-scaled-object-dev
Namespace:                                                                               chili-dev
Labels:                                                                                  app.kubernetes.io/managed-by=keda-operator
                                                                                         app.kubernetes.io/name=keda-hpa-kedaloganalytics-consumer-scaled-object-dev
                                                                                         app.kubernetes.io/part-of=kedaloganalytics-consumer-scaled-object-dev
                                                                                         app.kubernetes.io/version=2.7.0
                                                                                         deploymentName=keda-consumer-dev
                                                                                         scaledobject.keda.sh/name=kedaloganalytics-consumer-scaled-object-dev
Annotations:                                                                             <none>
CreationTimestamp:                                                                       Tue, 10 May 2022 14:18:20 +0200
Reference:                                                                               Deployment/chiliservice-dev
Metrics:                                                                                 ( current / target )
  "s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d" (target average value):  0 / 1
Min replicas:                                                                            1
Max replicas:                                                                            10
Deployment pods:                                                                         1 current / 1 desired
Conditions:
  Type            Status  Reason                   Message
  ----            ------  ------                   -------
  AbleToScale     True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive   False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric chili-dev/s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kedaloganalytics-consumer-scaled-object-dev,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d
  ScalingLimited  True    TooFewReplicas           the desired replica count is less than the minimum replica count
Events:
  Type     Reason                   Age                    From                       Message
  ----     ------                   ----                   ----                       -------
  Normal   SuccessfulRescale        54m (x2 over 88m)      horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  Warning  FailedGetExternalMetric  4m7s (x334 over 4h9m)  horizontal-pod-autoscaler  unable to get external metric chili-dev/s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: kedaloganalytics-consumer-scaled-object-dev,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d

Describe of the scaled object:

Name:         kedaloganalytics-consumer-scaled-object-dev
Namespace:    chili-dev
Labels:       deploymentName=keda-consumer-dev
              scaledobject.keda.sh/name=kedaloganalytics-consumer-scaled-object-dev
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2022-05-10T12:18:18Z
  Finalizers:
    finalizer.keda.sh
  Generation:  3
  Managed Fields:
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:deploymentName:
      f:spec:
        .:
        f:cooldownPeriod:
        f:maxReplicaCount:
        f:minReplicaCount:
        f:pollingInterval:
        f:scaleTargetRef:
          .:
          f:name:
        f:triggers:
    Manager:      HashiCorp
    Operation:    Update
    Time:         2022-05-10T12:18:18Z
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizer.keda.sh":
        f:labels:
          f:scaledobject.keda.sh/name:
      f:status:
        .:
        f:conditions:
        f:externalMetricNames:
        f:lastActiveTime:
        f:originalReplicaCount:
        f:scaleTargetGVKR:
          .:
          f:group:
          f:kind:
          f:resource:
          f:version:
        f:scaleTargetKind:
    Manager:      keda
    Operation:    Update
    Time:         2022-05-10T14:50:59Z
    API Version:  keda.sh/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:health:
          .:
          f:s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d:
            .:
            f:numberOfFailures:
            f:status:
    Manager:         keda-adapter
    Operation:       Update
    Time:            2022-05-10T14:51:14Z
  Resource Version:  2871768
  UID:               cf548230-b482-4ba6-affa-305c7ed6cf4a
Spec:
  Cooldown Period:    30
  Max Replica Count:  10
  Min Replica Count:  1
  Polling Interval:   30
  Scale Target Ref:
    Name:  chiliservice-dev
  Triggers:
    Authentication Ref:
      Name:  trigger-keda-dev
    Metadata:
      Query:  let tenant = "dev";
AppMetrics
| where TimeGenerated > ago(5m)
| where Name == "OverdueTasks"
| where AppRoleInstance contains tenant
| summarize by OverdueTasks_Sum = Sum

      Threshold:  1
    Type:         azure-log-analytics
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is performed because triggers are active
    Reason:   ScalerActive
    Status:   True
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
  External Metric Names:
    s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d
  Health:
    s0-azure-log-analytics-2c590a65-54c2-45a2-9aaf-b529fa0b4b3d:
      Number Of Failures:  0
      Status:              Happy
  Last Active Time:        2022-05-10T16:32:26Z
  Original Replica Count:  2
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:               <none>

Errors from KEDA operator :

1.6521939320277655e+09 ERROR scalehandler Error getting scale decision {"scaledobject.Name": "kedaloganalytics-consumer-scaled-object-dev", "scaledObject.Namespace": "chili-dev", "scaleTarget.Name": "chiliservice-dev", "error": "failed to execute IsActive function. Scaled object: kedaloganalytics-consumer-scaled-object-dev. Namespace: chili-dev. Inner Error: error calling Log Analytics REST api. Inner Error: Post \"https://api.loganalytics.io/v1/workspaces/2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"} github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers /workspace/pkg/scaling/scale_handler.go:278 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop /workspace/pkg/scaling/scale_handler.go:149

Expected Behavior

No failure event on the HPA.

Actual Behavior

Warning FailedGetExternalMetric

Steps to Reproduce the Problem

Install KEDA with helm chart on AKS cluster.
Configure the scaler following: https://keda.sh/docs/2.7/scalers/azure-log-analytics/

Logs from KEDA operator

I0510 12:09:48.928119       1 request.go:665] Waited for 1.045989048s due to client-side throttling, not priority and fairness, request: GET:https://10.2.0.1:443/apis/autoscaling/v1?timeout=32s
1.6521845901325662e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6521845901359537e+09	INFO	setup	Running on Kubernetes 1.21	{"version": "v1.21.9"}
1.652184590136202e+09	INFO	setup	Starting manager
1.6521845901362207e+09	INFO	setup	KEDA Version: 2.7.0
1.6521845901362245e+09	INFO	setup	Git Commit: 5f8af5cd90805d8b9c5964e1619a8b2bdcec6cfd
1.6521845901362278e+09	INFO	setup	Go Version: go1.17.3
1.652184590136231e+09	INFO	setup	Go OS/Arch: linux/amd64
I0510 12:09:50.136603       1 leaderelection.go:248] attempting to acquire leader lease keda/operator.keda.sh...
1.6521845901368594e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.652184590136948e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
I0510 12:10:05.290087       1 leaderelection.go:258] successfully acquired lease keda/operator.keda.sh
1.652184605290749e+09	INFO	controller.scaledobject	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
1.6521846052907484e+09	INFO	controller.triggerauthentication	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
1.6521846052911603e+09	INFO	controller.triggerauthentication	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication"}
1.652184605290868e+09	INFO	controller.clustertriggerauthentication	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
1.6521846052912364e+09	INFO	controller.clustertriggerauthentication	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ClusterTriggerAuthentication"}
1.652184605290804e+09	INFO	controller.scaledjob	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
1.6521846052914548e+09	INFO	controller.scaledjob	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob"}
1.6521846052910922e+09	INFO	controller.scaledobject	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "source": "kind source: *v2beta2.HorizontalPodAutoscaler"}
1.6521846052915452e+09	INFO	controller.scaledobject	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject"}
1.6521846053933072e+09	INFO	controller.scaledjob	Starting workers	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob", "worker count": 1}
1.652184605393311e+09	INFO	controller.triggerauthentication	Starting workers	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication", "worker count": 1}
1.6521846053933315e+09	INFO	controller.scaledobject	Starting workers	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "worker count": 5}
1.652184605393649e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev"}
1.6521882252631824e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev"}
1.6521882548151696e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev"}
1.6521939020265949e+09	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "kedaloganalytics-consumer-scaled-object-dev", "scaledObject.Namespace": "chili-dev", "scaleTarget.Name": "chiliservice-dev", "error": "failed to execute IsActive function. Scaled object: kedaloganalytics-consumer-scaled-object-dev. Namespace: chili-dev. Inner Error: error calling Log Analytics REST api. Inner Error: Post \"https://api.loganalytics.io/v1/workspaces/2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
1.6521939021789622e+09	INFO	controller.scaledobject	Updated HPA according to ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev", "HPA.Namespace": "chili-dev", "HPA.Name": "keda-hpa-kedaloganalytics-consumer-scaled-object-dev"}
1.6521939162685077e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev"}
1.6521939193038576e+09	INFO	controller.scaledobject	Updated HPA according to ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev", "HPA.Namespace": "chili-dev", "HPA.Name": "keda-hpa-kedaloganalytics-consumer-scaled-object-dev"}
1.652193919318518e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev"}
1.6521939223443966e+09	INFO	controller.scaledobject	Updated HPA according to ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev", "HPA.Namespace": "chili-dev", "HPA.Name": "keda-hpa-kedaloganalytics-consumer-scaled-object-dev"}
1.6521939320277655e+09	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "kedaloganalytics-consumer-scaled-object-dev", "scaledObject.Namespace": "chili-dev", "scaleTarget.Name": "chiliservice-dev", "error": "failed to execute IsActive function. Scaled object: kedaloganalytics-consumer-scaled-object-dev. Namespace: chili-dev. Inner Error: error calling Log Analytics REST api. Inner Error: Post \"https://api.loganalytics.io/v1/workspaces/2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149
1.6521939613691688e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kedaloganalytics-consumer-scaled-object-dev", "namespace": "chili-dev"}
1.6521939620284812e+09	ERROR	scalehandler	Error getting scale decision	{"scaledobject.Name": "kedaloganalytics-consumer-scaled-object-dev", "scaledObject.Namespace": "chili-dev", "scaleTarget.Name": "chiliservice-dev", "error": "failed to execute IsActive function. Scaled object: kedaloganalytics-consumer-scaled-object-dev. Namespace: chili-dev. Inner Error: error calling Log Analytics REST api. Inner Error: Post \"https://api.loganalytics.io/v1/workspaces/2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/query\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:278
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:149

KEDA Version

2.7.0

Kubernetes Version

1.21

Platform

Microsoft Azure

Scaler Details

Azure Log Analytics

Anything else?

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 26 (10 by maintainers)

Most upvoted comments

the error says prometheus query %s returned multiple elements. I guess that the error here is clear, but the docs isn’t clear enough, but we could modify the message clarifying the limitation in code too. Maybe something like prometheus query %s returned multiple elements but only single element is supported

JorTurFer on Sep 15, 2022

IDK, I’d say that they are different errors because errors in the upstreams produce the same behaviour. @Anoojak, could you share your metrics-server logs?

The original issue was a timeout doing the request Inner Error: Post \"https://api.loganalytics.io/v1/workspaces/2c590a65-54c2-45a2-9aaf-b529fa0b4b3d/query\": context deadline exceeded.

I mean, the consequence is the same, but I think the root cause it’s different, let’s wait till logs

JorTurFer on Sep 14, 2022

hi @kibnelbachyr I have checked them and I can’t find anything (but I’m not an expert in Log Analytics). Maybe @v-shenoy can take a look if he has time

JorTurFer on Jun 8, 2022