keda: New Relic metric shows incorrect value

Report

I’m having issues using New Relic queries properly. The current value shown in HPA does not match the response in New Relic. The reported value is too high, and the pods are stuck at their maximum capacity.

I went over previous GitHub issues/discussions about the same problem, and unfortunately, the documentation/FAQs didn’t help in my case: https://github.com/kedacore/keda/issues/2740 https://github.com/kedacore/keda/discussions/2451

Expected Behavior

I expect the behavior to match the query result in New Relic and the HPA to scale my deployment according to the query results.

Actual Behavior

The query I’m running: SELECT rate(count(apm.service.transaction.duration), 1 minute) FROM Metric WHERE appName = 'my-app-name' TIMESERIES

HPA output: NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE keda-my-app Deployment/my-app 340141296/50 (avg) 1 5 5 3h56m

As you can see from the graph - the value of the NRQL is around 100 - because I’m using the default average behavior it should be ~20 (divided by 5 pods) which is well below the threshold (50).

I still can’t figure out where the value of 340141296 came from, but because of this high value, the HPA is always configured for the maximum amount of replicas.

Steps to Reproduce the Problem

Add a New Relic scaler with a simple APM query for RPM (requests per minute)

Logs from KEDA operator

2023-11-23T19:50:37Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-app-name","namespace":"default"}, "namespace": "default", "name": "my-app-name", "reconcileID": "2db87bd4-7cb1-4728-ae5f-e753c8131557"}
2023-11-23T19:50:52Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-app-name","namespace":"default"}, "namespace": "default", "name": "my-app-name", "reconcileID": "d35a43b5-456f-4e8c-9811-eb07ee961220"}
2023-11-23T19:50:52Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-app-name","namespace":"default"}, "namespace": "default", "name": "my-app-name", "reconcileID": "a511b801-08b4-40a0-84f4-b09ba845b6b7"}
2023-11-23T19:51:09Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-app-name","namespace":"default"}, "namespace": "default", "name": "my-app-name", "reconcileID": "4059d985-c9d4-45b6-8f0a-1d32464ac682"}
2023-11-23T19:51:25Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-app-name","namespace":"default"}, "namespace": "default", "name": "my-app-name", "reconcileID": "a274d10e-daa8-4507-9d4f-317b79e55488"}
2023-11-23T19:52:10Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-app-name","namespace":"default"}, "namespace": "default", "name": "my-app-name", "reconcileID": "36c77231-eb44-4e0a-89cf-1aa049832a43"}

Scaled object status:

Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is performed because triggers are active
    Reason:   ScalerActive
    Status:   True
    Type:     Active
    Message:  No fallbacks are active on this scaled object
    Reason:   NoFallbackFound
    Status:   False
    Type:     Fallback
    Status:   Unknown
    Type:     Paused

KEDA Version

2.12.0

Kubernetes Version

1.27

Platform

Amazon Web Services

Scaler Details

New Relic

Anything else?

No response