keda: Using Hashicorp Vault secrets with TriggerAuthentication results in unable to convert Vault Data value error

Report

Configuring a TriggerAuthentication object to use hashiCorpVault to get the values for various parameters on a ScaledObject. The scale_resolvers returns unable to convert Vault Data value

Expected Behavior

The result of the queryKey to be used for the new-relic scaler

Actual Behavior

The following error related to the new-relic scaler

keda-operator-ddd8757f-9bnnv keda-operator 1.6449390228145654e+09	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "newrelic-ta-scaledobject", "error": "unable to convert Vault Data value"}  

Steps to Reproduce the Problem

1. Create kind cluster kind create cluster --name keda-test
2. kubectl create namespace keda
3. helm install keda kedacore/keda --namespace keda
4. Setup vault server: vault server -dev -dev-root-token-id="root" -dev-listen-address=0.0.0.0:8200 >> /dev/null &
5. export VAULT_ADDR='http://0.0.0.0:8200'
6. vault login root
7. export VAULT_SA_NAME=$(kubectl get sa keda-operator -n keda --output jsonpath="{.secrets[*]['name']}")
8. export SA_JWT_TOKEN=$(kubectl get secret -n keda $VAULT_SA_NAME --output 'go-template={{ .data.token }}' | base64 --decode)
9. export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
10. export K8S_HOST=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}')
11. vault auth enable kubernetes
12. Write vault config

vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="$K8S_HOST" \
        kubernetes_ca_cert="$SA_CA_CRT" \
        issuer="https://kubernetes.default.svc.cluster.local"

vault write auth/kubernetes/role/keda \
        bound_service_account_names=keda-operator \
        bound_service_account_namespaces=keda \
        policies=keda \
        ttl=24h

1. Create the TriggerAuthentication object triggerauthentication.yaml

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-vault
spec:
  hashiCorpVault:
    address: http://192.168.0.4:8200
    authentication: kubernetes
    role: keda
    mount: kubernetes
    credential:
      serviceAccount: /var/run/secrets/kubernetes.io/serviceaccount/token
    secrets:
    - parameter: queryKey
      key: keda-nr-key
      path: /kv-v1/keda/secret

1. Create ScaledObject scaledobject.yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: newrelic-ta-scaledobject
spec:
  scaleTargetRef:
    name: deployment-nr-ta
  minReplicaCount: 1
  maxReplicaCount: 50
  cooldownPeriod: 5
  idleReplicaCount: 0
  triggers:
  - type: new-relic
    metadata:
      account: '1234567'
      region: "US"
      noDataError: "true"
      nrql: "SELECT latest(allocatablePods) from K8sNodeSample WHERE clusterName = 'cluster-name'"
      threshold: '10'
    authenticationRef:
     name: keda-trigger-auth-vault

1. Create deployment object deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-nr-ta
spec:
  replicas: 0
  selector:
    matchLabels:
      app: deployment-nr-ta
  template:
    metadata:
      labels:
        app: deployment-nr-ta
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            name: deployment-nr-ta
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            name: deployment-nr-ta
      terminationGracePeriodSeconds: 0
      containers:
        - name: pause-deployment-nr-ta
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          resources:
            requests:
              cpu: 250m
              memory: 250m

1. Create namespace kubectl create ns keda-test
2. Apply deployment kubectl apply -f deployment.yaml -n keda-test
3. Apply TriggerAuthentication object kubectl apply -f triggerauthentication.yaml -n keda-test
4. Apply ScaledObject kubectl apply -f scaledobject.yaml -n keda-test
5. Review the logs using stern or kubectl logs

Logs from KEDA operator

keda-operator-ddd8757f-9bnnv keda-operator 1.6449390227989428e+09	INFO	controller.scaledobject	Creating a new HPA	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "newrelic-ta-scaledobject", "namespace": "keda-test", "HPA.Namespace": "keda-test", "HPA.Name": "keda-hpa-newrelic-ta-scaledobject"}
keda-operator-ddd8757f-9bnnv keda-operator 1.6449390228145654e+09	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "newrelic-ta-scaledobject", "error": "unable to convert Vault Data value"}    

KEDA Version

2.6.0

Kubernetes Version

v1.21.1

Platform

Other

Scaler Details

• new-relic and rabbitmq

Anything else?

Create Vault v1 secret vault secrets enable -path="kv-v1" -description="Test V1" kv Added NR key vault kv put kv-v1/keda/secret keda-nr-key=NRAK-12345678901234 Also tested using a v2 and got same error.