keda: Deployment is not scaling down when using podidentity for Azure Eventhubs

Report

We have created a java app on aks based on eventhub trigger, and we are using keda for autoscaling.

When we used the connection strings directly in the deployments the autoscaling and scale down worked properly for eventhub trigger. But when we used the podidentity for scaled object, it is scaling up but scaling down is not happening and the HPA metric is showing higher number even after all the checkpoints are done.

find the below yml files

Deployment.yml

kind: Deployment
metadata:
  name: events-processor-deployment
spec:
  selector:
    matchLabels:
      app: events-processor
  template:
    metadata:
      labels:
        app: events-processor
        # aadpodidbinding: function-apps-binding
    spec:
    
      containers:
      - name: events-processor-container
        image: kubepocmyacr.azurecr.io/sample-image:1.0
        env:
        - name: BLOB_STORAGE_CONNECTION
          value: <Placeholder Connection String>
        # JAVA OPTS
        - name: JAVA_OPTS
          value: "-XX:MaxRAMPercentage=75.0"
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "2.5Gi"
            cpu: "1500m"`

TriggerAuthentication.yml

kind: TriggerAuthentication
metadata:
  name: azure-eventhub-trigger-auth
  namespace: keda-system

spec:
  podIdentity:
    provider: azure

ScaledObject.yml

kind: ScaledObject
metadata:
  name: azure-eventhub-spring-scaledobject
  namespace: keda-system
spec:
  scaleTargetRef:
    name: events-processor-deployment
  minReplicaCount: 1
  maxReplicaCount: 5
  cooldownPeriod: 30
  pollingInterval: 15
  triggers:
  - type: azure-eventhub
    metadata:
      # Required
      storageConnectionFromEnv: BLOB_STORAGE_CONNECTION
      eventHubNamespace: poceventhubcosmos
      eventHubName: demo-analysis-eventhub
      consumerGroup: $Default
      unprocessedEventThreshold: '50'
      checkpointStrategy: 'blobMetadata'
      blobContainer: 'kubernetes-poc'
    authenticationRef:
      name: azure-eventhub-trigger-auth

### Expected Behavior

Keda should be able to scale down the deployment even when we use pod identity

### Actual Behavior

Keda is not able to scale down the deployment when using podidentity, even though we are checkpoint every message, And the HPA metrics are showing higher number.



### Steps to Reproduce the Problem


This is working as expected when we use below metadata in scaledobject

metadata:
  storageConnectionFromEnv: BLOB_STORAGE_CONNECTION
  connectionFromEnv: EVENTHUB_CONNECTION
  consumerGroup: $Default
  unprocessedEventThreshold: '50'
  checkpointStrategy: 'blobMetadata'
  blobContainer: 'kubernetes-poc'

But when we use the below metadata for pod identity, its not working as expected

metadata:
  storageConnectionFromEnv: BLOB_STORAGE_CONNECTION
  eventHubNamespace: poceventhubcosmos
  eventHubName: demo-analysis-eventhub
  consumerGroup: $Default
  unprocessedEventThreshold: '50'
  checkpointStrategy: 'blobMetadata'
  blobContainer: 'kubernetes-poc'
authenticationRef:
  name: azure-eventhub-trigger-auth

### Logs from KEDA operator

Keda operator logs

1.648033250782659e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.648033250782781e+09 INFO controller.scaledobject Creating a new HPA {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system", "HPA.Namespace": "keda-system", "HPA.Name": "keda-hpa-azure-eventhub-spring-scaledobject"}

1.648033251146449e+09 INFO controller.scaledobject Initializing Scaling logic according to ScaledObject Specification {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332511549592e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332557244997e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332696407406e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}

1.6480332853822763e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "azure-eventhub-spring-scaledobject", "namespace": "keda-system"}




Keda operator metrics logs

2022-03-23T16:31:09+05:30 I0323 11:01:09.589547 1 trace.go:205] Trace[937030477]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-system/s0-azure-eventhub-$default,user-agent:kube-controller-manager/v1.21.9 (linux/amd64) kubernetes/37f338a/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:35c82ed1-8f1f-4669-a9a9-b12b90ed2930,client:172.31.58.69,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (23-Mar-2022 11:01:06.213) (total time: 3375ms):

2022-03-23T16:31:09+05:30 Trace[937030477]: ---"Listing from storage done" 3375ms (11:01:09.589)

2022-03-23T16:31:09+05:30 Trace[937030477]: [3.375537885s] [3.375537885s] END

2022-03-23T16:31:25+05:30 I0323 11:01:25.367790 1 trace.go:205] Trace[407449632]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-system/s0-azure-eventhub-$default,user-agent:kube-controller-manager/v1.21.9 (linux/amd64) kubernetes/37f338a/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:4cc1c7be-41c0-43a2-9113-77324ca64220,client:172.31.58.69,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (23-Mar-2022 11:01:24.691) (total time: 676ms):

2022-03-23T16:31:25+05:30 Trace[407449632]: ---"Listing from storage done" 676ms (11:01:25.367)

2022-03-23T16:31:25+05:30 Trace[407449632]: [676.120576ms] [676.120576ms] END

2022-03-23T16:31:40+05:30 I0323 11:01:40.982286 1 trace.go:205] Trace[1154341616]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/keda-system/s0-azure-eventhub-$default,user-agent:kube-controller-manager/v1.21.9 (linux/amd64) kubernetes/37f338a/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:0d0f7cf8-e718-4333-b5e8-3cef73bb8eca,client:172.31.58.69,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (23-Mar-2022 11:01:40.421) (total time: 560ms):

2022-03-23T16:31:40+05:30 Trace[1154341616]: ---"Listing from storage done" 560ms (11:01:40.982)


### KEDA Version

2.6.1

### Kubernetes Version

1.21

### Platform

Microsoft Azure

### Scaler Details

Azure Eventhub

### Anything else?

We think that it is not able to use the storage connection environment variable when we use pod identity for eventhub credentials

We followed below documentation for implementing this:
https://keda.sh/docs/2.6/scalers/azure-event-hub/