keda: KEDA scaler not working on AKS with trigger authentication using pod identity

Report

KEDA scaler not scales with scaled object defined with trigger using pod identity for authentication for service bus queue. I’m following this KEDA service bus triggered scaling project.
The scaling works fine with the connection string, but when I try to scale using the pod identity for KEDA scaler the keda operator fails to get the azure identity bound to it with the following keda operator error message log:

github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
        /workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:145
2021-10-10T17:35:53.916Z        ERROR   azure_servicebus_scaler error   {"error": "failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}\n"}

My scaler objects’ definition is as below:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-service-bus-orders
spec:
  podIdentity:
    provider: azure
---
apiVersion: keda.sh/v1alpha1 
kind: ScaledObject
metadata:
  name: order-scaler
spec:
  scaleTargetRef:
    name: order-processor
  # minReplicaCount: 0 Change to define how many minimum replicas you want
  maxReplicaCount: 10
  triggers:
  - type: azure-servicebus
    metadata:
      namespace: demodemobus
      queueName: orders
      messageCount: '5'
    authenticationRef:
      name: trigger-auth-service-bus-orders

Im deploying the azure identity to the namespace keda where my keda deployment resides. And installs KEDA with the following command to set the pod identity binding using helm:

helm install keda kedacore/keda --set podIdentity.activeDirectory.identity=app-autoscaler --namespace keda

Expected Behavior

The KEDA scaler should have worked fine with the assigned pod identity and access token to perform scaling

Actual Behavior

The KEDA operator could not be able to find the azure identity assigned and scaling fails

Steps to Reproduce the Problem

Create the azure identity and bindings for the KEDA
Install KEDA with the aadpodidentitybinding
Create the scaledobject and triggerauthentication using KEDA pod identity
The scaler fails to authenticate and scale

Logs from KEDA operator

github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
        /workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:145
2021-10-10T17:41:54.909Z        ERROR   azure_servicebus_scaler error   {"error": "failed to refresh token, error: adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}\n"}

KEDA Version

No response

Kubernetes Version

1.20

Platform

Microsoft Azure

Scaler Details

Azure Service Bus

Anything else?

No response

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 48 (16 by maintainers)

Most upvoted comments

Posting e2e Script for AKS with kubenet plugin, if it is of any assistance to some:

# Define aks name and resource group
$aksResourceGroup = "K8sScalingDemo"
$aksName = "K8sScalingDemo"

# Create resource group
az group create -n $aksResourceGroup -l centralindia

# Create the aks cluster with default kubenet plugin
az aks create -n $aksName -g $aksResourceGroup

# Resourcegroup where the aks resources will be deployed
$resourceGroup = "$(az aks show -g $aksResourceGroup -n $aksName --query nodeResourceGroup -otsv)"

# Set the kubectl context to the newly created aks cluster
az aks get-credentials -n $aksName -g $aksResourceGroup

# Install AAD Pod Identity into the aad-pod-identity namespace using helm
kubectl create namespace aad-pod-identity
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install aad-pod-identity aad-pod-identity/aad-pod-identity --namespace aad-pod-identity

# Check the status of installation 
kubectl --namespace=aad-pod-identity get pods -l "app.kubernetes.io/component=mic"
kubectl --namespace=aad-pod-identity get pods -l "app.kubernetes.io/component=nmi"

# the nmi components will Crashloop, ignore them for now. We will make them right later

# Get Resourcegroup Id of our $ResourceGroup
$resourceGroup_ResourceId = az group show --name $resourceGroup --query id -otsv

# Get the aks cluster kubeletidentity client id
$aad_pod_identity_clientid = az aks show -g $aksResourceGroup -n $aksName --query identityProfile.kubeletidentity.clientId -otsv

# Assign required roles for cluster over the resourcegroup
az role assignment create --role "Managed Identity Operator" --assignee $aad_pod_identity_clientid  --scope $resourceGroup_ResourceId
az role assignment create --role "Virtual Machine Contributor" --assignee $aad_pod_identity_clientid  --scope $resourceGroup_ResourceId

# Create autoscaler azure identity and get client id and resource id of the autoscaler identity
$autoScaleridentityName = "autoscaler-aad-identity"
az identity create --name $autoScaleridentityName  --resource-group $resourceGroup
$autoscaler_aad_identity_clientId = az identity show --name $autoScaleridentityName  --resource-group $resourceGroup --query clientId -otsv
$autoscaler_aad_identity_resourceId = az identity show --name $autoScaleridentityName  --resource-group $resourceGroup --query id -otsv

# Create the app azure identity and get client id and resource id of the app identity
$appIdentityName = "app-aad-identity"
az identity create --name app-aad-identity --resource-group $resourceGroup
$app_aad_identity_clientId = az identity show --name $appIdentityName --resource-group $resourceGroup --query clientId -otsv
$app_aad_identity_resourceId = az identity show --name $appIdentityName --resource-group $resourceGroup --query id -otsv

# Create service bus and queue
$servicebus = 'svcbusdemo'
az servicebus namespace create --name $servicebus --resource-group $resourceGroup --sku basic
$servicebus_namespace_resourceId = az servicebus namespace show --name $servicebus --resource-group $resourceGroup --query id -otsv

az servicebus queue create --namespace-name $servicebus --name orders --resource-group $resourceGroup
$servicebus_queue_resourceId = az servicebus queue show --namespace-name $servicebus --name orders --resource-group $resourceGroup --query id -otsv

# Assign Service Bus Data Receiver role to the app identity created
az role assignment create --role 'Azure Service Bus Data Receiver' --assignee $app_aad_identity_clientId  --scope $servicebus_queue_resourceId

# Create a namespace for order app deployment
kubectl create namespace keda-dotnet-sample

# Create a yaml deployment configuration variable
$app_with_identity_yaml= @"
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: $appIdentityName
  annotations:
    aadpodidentity.k8s.io/Behavior: namespaced
spec:
  type: 0 # 0 means User-assigned MSI
  resourceID: $app_aad_identity_resourceId
  clientID: $app_aad_identity_clientId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: $appIdentityName-binding
spec:
  azureIdentity: $appIdentityName
  selector: order-processor
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-processor
  labels:
    app: order-processor
spec:
  selector:
    matchLabels:
      app: order-processor
  template:
    metadata:
      labels:
        app: order-processor
        aadpodidbinding: order-processor
    spec:
      containers:
      - name: order-processor
        image: ghcr.io/kedacore/sample-dotnet-worker-servicebus-queue:latest
        env:
        - name: KEDA_SERVICEBUS_AUTH_MODE
          value: ManagedIdentity
        - name: KEDA_SERVICEBUS_HOST_NAME
          value: $servicebus.servicebus.windows.net
        - name: KEDA_SERVICEBUS_QUEUE_NAME
          value: orders
        - name: KEDA_SERVICEBUS_IDENTITY_USERASSIGNEDID
          value: $app_aad_identity_clientId
"@

# Create the app deployment with identity bindings using kubectl apply
$app_with_identity_yaml | kubectl apply --namespace keda-dotnet-sample -f -

# Now the order processor app works with the pod identity and 
# processes the queues 
# You can refer the [project ](https://github.com/kedacore/sample-dotnet-worker-servicebus-queue/blob/main/pod-identity.md) for that.

# Now start installation of KEDA in namespace keda-system

kubectl create namespace keda-system

# Create a pod identity and binding for autoscaler azure identity
$autoscaler_yaml =@"
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: $autoScaleridentityName
spec:
  type: 0 # 0 means User-assigned MSI
  resourceID: $autoscaler_aad_identity_resourceId
  clientID: $autoscaler_aad_identity_clientId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: $autoScaleridentityName-binding
spec:
  azureIdentity: $autoScaleridentityName
  selector: $autoScaleridentityName
"@
$autoscaler_yaml | kubectl apply --namespace keda-system -f -

# Install KEDA using helm
helm install keda kedacore/keda --set podIdentity.activeDirectory.identity=autoscaler-aad-identity --namespace keda-system

# Assign Service Bus Data Owner role to keda autoscaler identity
az role assignment create --role 'Azure Service Bus Data Owner' --assignee $autoscaler_aad_identity_clientId --scope $servicebus_namespace_resourceId

# Apply scaled object definition and trigger authentication provider as `azure`
$aap_autoscaling_yaml = @"
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-service-bus-orders
spec:
  podIdentity:
    provider: azure
---
apiVersion: keda.sh/v1alpha1 
kind: ScaledObject
metadata:
  name: order-scaler
spec:
  scaleTargetRef:
    name: order-processor
  # minReplicaCount: 0 Change to define how many minimum replicas you want
  maxReplicaCount: 10
  triggers:
  - type: azure-servicebus
    metadata:
      namespace: $servicebus
      queueName: orders
      messageCount: '5'
    authenticationRef:
      name: trigger-auth-service-bus-orders
"@

$aap_autoscaling_yaml | kubectl apply --namespace keda-dotnet-sample -f -

# Now the Keda is getting 401 unauthorized error as the AAD Pod Identity comnponent `nmi` is not runnig on the system
# To fix it edit the daemonset for `nmi` component
# add the container arg `--allow-network-plugin-kubenet=true` by editing the `daemonset.apps/aad-pod-identity-nmi`
kubectl edit daemonset.apps/aad-pod-identity-nmi -n aad-pod-identity

# the containe arg section should look like this after editing:
    spec:
      containers:
      - args:
        - --node=$(NODE_NAME)
        - --http-probe-port=8085
        - --enableScaleFeatures=true
        - --metadata-header-required=true
        - --operation-mode=standard
        - --kubelet-config=/etc/default/kubelet
        - --allow-network-plugin-kubenet=true
        env:

# Now the KEDA is authenticated by aad-pod-identity metadata endpoint and the orderapp should scale up 
# with the queue counts
# If the order app still falls back to errors please delete and redeploy it.
# And that's it you just scaled your app up using KEDA on Kubenet AKS cluster.

Note: Read this instruction before you run AAD Identity On a Kubenet powered AKS.

iarunpaul on Jan 24, 2022

@tomkerkhove @JorTurFer educated guess but 100% hit. selector is a problem there. I fixed and it works now. attaching e2e script it does everything except creating aks cluster. prvide aks name, aks rosource group name, node group name, and happy keda-ing 😃.

$resourceGroup = "MC_K8sScalingDemo_K8sScalingDemo_uksouth"
$aksResourceGroup = "K8sScalingDemo"
$aksName = "K8sScalingDemo"


kubectl create namespace aad-pod-identity
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install aad-pod-identity aad-pod-identity/aad-pod-identity --namespace aad-pod-identity

$resourceGroup_ResourceId = az group show --name $resourceGroup --query id -otsv
$aad_pod_identity_clientid = az aks show -g $aksResourceGroup -n $aksName --query identityProfile.kubeletidentity.clientId -otsv
az role assignment create --role "Managed Identity Operator" --assignee $aad_pod_identity_clientid  --scope $resourceGroup_ResourceId
az role assignment create --role "Virtual Machine Contributor" --assignee $aad_pod_identity_clientid  --scope $resourceGroup_ResourceId

$autoScaleridentityName = "autoscaler-aad-identity"
az identity create --name $autoScaleridentityName  --resource-group $resourceGroup
$autoscaler_aad_identity_clientId = az identity show --name $autoScaleridentityName  --resource-group $resourceGroup --query clientId -otsv
$autoscaler_aad_identity_resourceId = az identity show --name $autoScaleridentityName  --resource-group $resourceGroup --query id -otsv

$appIdentityName = "app-aad-identity"
az identity create --name app-aad-identity --resource-group $resourceGroup
$app_aad_identity_clientId = az identity show --name $appIdentityName --resource-group $resourceGroup --query clientId -otsv
$app_aad_identity_resourceId = az identity show --name $appIdentityName --resource-group $resourceGroup --query id -otsv

az servicebus namespace create --name svcbusdebe --resource-group $resourceGroup --sku basic
$servicebus_namespace_resourceId = az servicebus namespace show --name svcbusdebe --resource-group $resourceGroup --query id -otsv

az servicebus queue create --namespace-name svcbusdebe --name orders --resource-group $resourceGroup
$servicebus_queue_resourceId = az servicebus queue show --namespace-name svcbusdebe --name orders --resource-group $resourceGroup --query id -otsv

az role assignment create --role 'Azure Service Bus Data Receiver' --assignee $app_aad_identity_clientId  --scope $servicebus_queue_resourceId

kubectl create namespace keda-dotnet-sample

$app_with_identity_yaml= @"
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: $appIdentityName
  annotations:
    aadpodidentity.k8s.io/Behavior: namespaced
spec:
  type: 0 # 0 means User-assigned MSI
  resourceID: $app_aad_identity_resourceId
  clientID: $app_aad_identity_clientId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: $appIdentityName-binding
spec:
  azureIdentity: $appIdentityName
  selector: order-processor
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-processor
  labels:
    app: order-processor
spec:
  selector:
    matchLabels:
      app: order-processor
  template:
    metadata:
      labels:
        app: order-processor
        aadpodidbinding: order-processor
    spec:
      containers:
      - name: order-processor
        image: ghcr.io/kedacore/sample-dotnet-worker-servicebus-queue:latest
        env:
        - name: KEDA_SERVICEBUS_AUTH_MODE
          value: ManagedIdentity
        - name: KEDA_SERVICEBUS_HOST_NAME
          value: svcbusdebe.servicebus.windows.net
        - name: KEDA_SERVICEBUS_QUEUE_NAME
          value: orders
        - name: KEDA_SERVICEBUS_IDENTITY_USERASSIGNEDID
          value: $app_aad_identity_clientId
"@
$app_with_identity_yaml | kubectl apply --namespace keda-dotnet-sample -f -


kubectl create namespace keda-system

$autoscaler_yaml =@"
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: $autoScaleridentityName
spec:
  type: 0 # 0 means User-assigned MSI
  resourceID: $autoscaler_aad_identity_resourceId
  clientID: $autoscaler_aad_identity_clientId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: $autoScaleridentityName-binding
spec:
  azureIdentity: $autoScaleridentityName
  selector: $autoScaleridentityName
"@

$autoscaler_yaml | kubectl apply --namespace keda-system -f -

helm install keda kedacore/keda --set podIdentity.activeDirectory.identity=autoscaler-aad-identity --namespace keda-system

az role assignment create --role 'Azure Service Bus Data Owner' --assignee $autoscaler_aad_identity_clientId --scope $servicebus_namespace_resourceId


$aap_autoscaling_yaml = @"
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-service-bus-orders
spec:
  podIdentity:
    provider: azure
---
apiVersion: keda.sh/v1alpha1 
kind: ScaledObject
metadata:
  name: order-scaler
spec:
  scaleTargetRef:
    name: order-processor
  # minReplicaCount: 0 Change to define how many minimum replicas you want
  maxReplicaCount: 10
  triggers:
  - type: azure-servicebus
    metadata:
      namespace: svcbusdebe
      queueName: orders
      messageCount: '5'
    authenticationRef:
      name: trigger-auth-service-bus-orders
"@
$aap_autoscaling_yaml | kubectl apply --namespace keda-dotnet-sample -f -

dariuszbz on Jan 17, 2022

Hi @iarunpaul, First, please ignore the github-bot reaction, it’s an error in new workflow that we have merged a few days ago. We are working to fix it, but I’d like to apologize. Second, I will try to test it ASAP. Are you getting this error after some time or directly?

JorTurFer on Nov 6, 2021

Finally I found it! My set up is for a workaround with KEDA for kubenet interface. I followed the e2e script by @dariuszbz, where my cluster is aks with the default kubenet network plugin. The order application works fine without any nmi modifications and processes messages using the managed identity assigned to the application. But when installed KEDA using helm and set the pod identity, keda operator pod fails to get authorized: Logs are:

  2022-01-23T16:13:17.237Z        INFO    scaleexecutor   Successfully set ScaleTarget replicas count to ScaledObject minReplicaCount     {"scaledobject.Name": "order-scaler", "scaledObject.Namespace": "keda-dotnet-sample", "scaleTarget.Name": "order-processor", "Original Replicas Count": 1, "New Replicas Count": 0}
2022-01-23T16:13:31.982Z        INFO    controller.scaledobject Reconciling ScaledObject        {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "order-scaler", "namespace": "keda-dotnet-sample"}
2022-01-23T16:13:47.036Z        ERROR   azure_servicebus_scaler error   {"error": "request failed: 401 Unauthorized"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:261

Then I had to edit nmi command arguement to add --allow-network-plugin-kubenet=true and the nmi starts running…

Now the nmi pod reads:

I0123 16:55:43.377799       1 standard.go:73] no clientID or resourceID in request. keda-system/keda-operator-584cc6777-5xscs has been matched with azure identity keda-system/autoscaler-aad-identity
I0123 16:55:43.377857       1 standard.go:179] matched identityType:0 clientid:1c04##### REDACTED #####7168 resource:https://servicebus.azure.net/
I0123 16:55:43.386639       1 server.go:199] status (200) took 8880095 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.244.1.3
I0123 16:56:13.378683       1 standard.go:73] no clientID or resourceID in request. keda-system/keda-operator-584cc6777-5xscs has been matched with azure identity keda-system/autoscaler-aad-identity
I0123 16:56:13.378762       1 standard.go:179] matched identityType:0 clientid:1c04##### REDACTED #####7168 resource:https://servicebus.azure.net/
I0123 16:56:13.388151       1 server.go:199] status (200) took 9511587 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.244.1.3
I0123 16:56:43.379373       1 standard.go:73] no clientID or resourceID in request. keda-system/keda-operator-584cc6777-5xscs has been matched with azure identity keda-system/autoscaler-aad-identity
I0123 16:56:43.379431       1 standard.go:179] matched identityType:0 clientid:1c04##### REDACTED #####7168 resource:https://servicebus.azure.net/
I0123 16:56:43.389095       1 server.go:199] status (200) took 9787784 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.244.1.3

Still the order application doesn’t work. You need to delete and redeploy the application and wait for a few minutes… And thats it! Your application scales on KEDA metrics. Happy scaling!! Note: Always follow the mitigation steps and configure policies before enabling aad-pod-identity in a cluster with Kubenet.

iarunpaul on Jan 23, 2022

Tring everything I have assigned the `myAksCluster-agentpool` in the `$NodeResourceGroup` with the `Virtual Machine Contributor` and `Manage Identity Operator` roles on `$NodeResourceGroup` scope and now my KEDA trigger started working!!!

kubectl get deploy -n keda-dotnet-sample -o wide


NAME              READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS        IMAGES                                                          SELECTOR
order-processor   10/10   10           10          166m   order-processor   ghcr.io/kedacore/sample-dotnet-worker-servicebus-queue:latest   app=order-processor

Ofcourse my application is not consuming the messages but the happy moment is that the KEDA works with pod id in AKS!

But I have to recreate and confirm it.

iarunpaul on Jan 17, 2022

Hi @iarunpaul , What version of KEDA are you using? I’ll try to take a look at this the week

KEDA 2.4.0

iarunpaul on Nov 6, 2021