keda: azure-servicebus scaler erroring with blank error

I am using the latest KEDA v2 release and after setting up my scaled function and scaled object it outputs every 30 seconds a very non-descriptive error into the keda-operator container logs. My configuration as far as I’m aware is correct and even switching KEDA to use debug logging it still gives no useful information to help me solve this issue.

Expected Behavior

The scaler should work and not throw errors every 30 seconds, or at least output a meaningful error if I have broken the configuration somewhere

Actual Behavior

The scaler is erroring with a blank error message

Steps to Reproduce the Problem

  1. Add an Azure Function that uses a service bus subscription
  2. Add a scaled object with the correct parameters
  3. It fails to scale correctly

Logs from KEDA operator

2021-01-11T21:56:46.388585819Z  2021-01-11T21:56:46.388Z	ERROR	azure_servicebus_scaler	error	{"error": ""}
2021-01-11T21:56:46.388640720Z  github.com/go-logr/zapr.(*zapLogger).Error
2021-01-11T21:56:46.388652220Z  	/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
2021-01-11T21:56:46.388660020Z  github.com/kedacore/keda/pkg/scalers.(*azureServiceBusScaler).IsActive
2021-01-11T21:56:46.388668920Z  	/workspace/pkg/scalers/azure_servicebus_scaler.go:134
2021-01-11T21:56:46.388676420Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
2021-01-11T21:56:46.388683720Z  	/workspace/pkg/scaling/scale_handler.go:203
2021-01-11T21:56:46.388688520Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
2021-01-11T21:56:46.388694320Z  	/workspace/pkg/scaling/scale_handler.go:192
2021-01-11T21:56:46.388699120Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
2021-01-11T21:56:46.388703620Z  	/workspace/pkg/scaling/scale_handler.go:135
2021-01-11T21:56:46.388708320Z  2021-01-11T21:56:46.388Z	DEBUG	scalehandler	Error getting scale decision	{"Error": ""}
2021-01-11T21:56:47.341918395Z  2021-01-11T21:56:47.341Z	ERROR	azure_servicebus_scaler	error	{"error": ""}
2021-01-11T21:56:47.341970495Z  github.com/go-logr/zapr.(*zapLogger).Error
2021-01-11T21:56:47.341981895Z  	/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
2021-01-11T21:56:47.341991695Z  github.com/kedacore/keda/pkg/scalers.(*azureServiceBusScaler).IsActive
2021-01-11T21:56:47.341999495Z  	/workspace/pkg/scalers/azure_servicebus_scaler.go:134
2021-01-11T21:56:47.342007396Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
2021-01-11T21:56:47.342013196Z  	/workspace/pkg/scaling/scale_handler.go:203
2021-01-11T21:56:47.342020596Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
2021-01-11T21:56:47.342026896Z  	/workspace/pkg/scaling/scale_handler.go:192
2021-01-11T21:56:47.342033296Z  github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
2021-01-11T21:56:47.342038996Z  	/workspace/pkg/scaling/scale_handler.go:135
2021-01-11T21:56:47.342044596Z  2021-01-11T21:56:47.341Z	DEBUG	scalehandler	Error getting scale decision	{"Error": ""}
2021-01-11T21:56:47.346960941Z  2021-01-11T21:56:47.346Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "attachment-thmbgn-funcs-scaledobject", "scaledObject.Namespace": "attachment", "scaleTarget.Name": "attachment-thmbgn-funcs-scaled"}

Specifications

  • KEDA Version: docker.io/kedacore/keda:2.0.0
  • Platform & Version: Linux, Azure Functions v3
  • Kubernetes Version: v.19.3
  • Scaler(s): azure-servicebus

Definitions

Scaled Object:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: attachment-thmbgn-funcs-scaledobject
spec:
  scaleTargetRef:
    name: attachment-thmbgn-funcs-scaled
  triggers:
  - metadata:
      connectionFromEnv: AzureServiceBusPrimary
      subscriptionName: images-function
      topicName: attachments
    type: azure-servicebus
  - metadata:
      connectionFromEnv: AzureServiceBusPrimary
      subscriptionName: pdf-function
      topicName: attachments
    type: azure-servicebus

Deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: attachment-thmbgn-funcs-scaled
  labels:
    env: pr-7095
spec:
  replicas: 0
  selector:
    matchLabels:
      app: attachment-thmbgn-funcs-scaled
  template:
    metadata:
      labels:
        app: attachment-thmbgn-funcs-scaled
        env: pr-7095
    spec:
      containers:
        - name: attachment-thmbgn-funcs-scaled
          image: <censored>
          ports:
            - containerPort: 80
              protocol: TCP
          env:
            - name: APPINSIGHTS_INSTRUMENTATIONKEY
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: APPINSIGHTS_INSTRUMENTATIONKEY
            - name: AttachmentsConnectionString
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AttachmentsConnectionString
            - name: AzureServiceBusPrimary
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AzureServiceBusPrimary
            - name: AzureStorageBlob
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AzureStorageBlob
            - name: AzureWebJobsStorage
              valueFrom:
                secretKeyRef:
                  name: attachment-thmbgn-funcs-secret
                  key: AzureWebJobsStorage
            - name: ASPNETCORE_ENVIRONMENT
              value: pr-7095
            - name: ASPNETCORE_FORWARDEDHEADERS_ENABLED
              value: 'true'
            - name: AzureFunctionsJobHost__functions__0
              value: ImageThumbnailGenerator
            - name: AzureFunctionsJobHost__functions__1
              value: PdfThumbnailGenerator
            - name: AttachmentImageHeight
              value: '250'
            - name: AttachmentImageWidth
              value: '250'
            - name: BlobContainer
              value: attachments
            - name: FUNCTIONS_EXTENSION_VERSION
              value: ~3
            - name: FUNCTIONS_WORKER_RUNTIME
              value: dotnet
            - name: ImageHeight
              value: '250'
            - name: ImageSubscription
              value: images-function
            - name: ImageWidth
              value: '250'
            - name: ImagesContainer
              value: images
            - name: PdfSubscription
              value: pdf-function
            - name: ThumbnailBlobContainer
              value: thumbnails
            - name: TopicName
              value: attachments
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 25m
              memory: 200Mi
          livenessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 60
            timeoutSeconds: 5
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
  strategy:
    type: Recreate

Secret (censored of course):

kind: Secret
apiVersion: v1
metadata:
  name: attachment-thmbgn-funcs-secret
data:
  APPINSIGHTS_INSTRUMENTATIONKEY: <censored-base64-value>
  AttachmentsConnectionString: >-
    <censored-base64-value>
  AzureServiceBusPrimary: >-
    <censored-base64-value>
  AzureStorageBlob: >-
    <censored-base64-value>
  AzureWebJobsStorage: >-
    <censored-base64-value>
type: Opaque

If you need any more information from me please do let me know, but I hope this is everything you need. Please note I have modified the definitions/logs very slightly just to remove sensitive company information but it all matches up.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (18 by maintainers)

Most upvoted comments

I’m afraid I’ll need @ahmelsayed for this

I don’t actually work at the company I was having these issues with anymore so I’d have to set a few things up but sure, happy to help. Simplest repo I know of is to try and use the Service Bus scaler using a key with only Send/Listen permissions (not Manage) which should be easy enough to try out.

Just checked the KEDA documentation and it says that you need a key with Manage permission so hopefully others won’t get caught up with this though, looks like it was in the docs when I was implementing this but apparently I can’t read 😄

@MattJeanes thanks for the investigation. We can keep this issue opened till https://github.com/Azure/azure-service-bus-go/issues/229 is resolved.

And if that issue won’t get fixed in foreseeable future, we can try to validate the connection string in KEDA.

Thanks for reporting, we’ll get to the bottom of this.

@ahmelsayed can you pick this up?