keda: azure-servicebus scaler erroring with blank error
I am using the latest KEDA v2 release and after setting up my scaled function and scaled object it outputs every 30 seconds a very non-descriptive error into the keda-operator container logs. My configuration as far as I’m aware is correct and even switching KEDA to use debug logging it still gives no useful information to help me solve this issue.
Expected Behavior
The scaler should work and not throw errors every 30 seconds, or at least output a meaningful error if I have broken the configuration somewhere
Actual Behavior
The scaler is erroring with a blank error message
Steps to Reproduce the Problem
- Add an Azure Function that uses a service bus subscription
- Add a scaled object with the correct parameters
- It fails to scale correctly
Logs from KEDA operator
2021-01-11T21:56:46.388585819Z 2021-01-11T21:56:46.388Z ERROR azure_servicebus_scaler error {"error": ""}
2021-01-11T21:56:46.388640720Z github.com/go-logr/zapr.(*zapLogger).Error
2021-01-11T21:56:46.388652220Z /go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
2021-01-11T21:56:46.388660020Z github.com/kedacore/keda/pkg/scalers.(*azureServiceBusScaler).IsActive
2021-01-11T21:56:46.388668920Z /workspace/pkg/scalers/azure_servicebus_scaler.go:134
2021-01-11T21:56:46.388676420Z github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
2021-01-11T21:56:46.388683720Z /workspace/pkg/scaling/scale_handler.go:203
2021-01-11T21:56:46.388688520Z github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
2021-01-11T21:56:46.388694320Z /workspace/pkg/scaling/scale_handler.go:192
2021-01-11T21:56:46.388699120Z github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
2021-01-11T21:56:46.388703620Z /workspace/pkg/scaling/scale_handler.go:135
2021-01-11T21:56:46.388708320Z 2021-01-11T21:56:46.388Z DEBUG scalehandler Error getting scale decision {"Error": ""}
2021-01-11T21:56:47.341918395Z 2021-01-11T21:56:47.341Z ERROR azure_servicebus_scaler error {"error": ""}
2021-01-11T21:56:47.341970495Z github.com/go-logr/zapr.(*zapLogger).Error
2021-01-11T21:56:47.341981895Z /go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
2021-01-11T21:56:47.341991695Z github.com/kedacore/keda/pkg/scalers.(*azureServiceBusScaler).IsActive
2021-01-11T21:56:47.341999495Z /workspace/pkg/scalers/azure_servicebus_scaler.go:134
2021-01-11T21:56:47.342007396Z github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScaledObjectScalers
2021-01-11T21:56:47.342013196Z /workspace/pkg/scaling/scale_handler.go:203
2021-01-11T21:56:47.342020596Z github.com/kedacore/keda/pkg/scaling.(*scaleHandler).checkScalers
2021-01-11T21:56:47.342026896Z /workspace/pkg/scaling/scale_handler.go:192
2021-01-11T21:56:47.342033296Z github.com/kedacore/keda/pkg/scaling.(*scaleHandler).startScaleLoop
2021-01-11T21:56:47.342038996Z /workspace/pkg/scaling/scale_handler.go:135
2021-01-11T21:56:47.342044596Z 2021-01-11T21:56:47.341Z DEBUG scalehandler Error getting scale decision {"Error": ""}
2021-01-11T21:56:47.346960941Z 2021-01-11T21:56:47.346Z DEBUG scaleexecutor ScaleTarget no change {"scaledobject.Name": "attachment-thmbgn-funcs-scaledobject", "scaledObject.Namespace": "attachment", "scaleTarget.Name": "attachment-thmbgn-funcs-scaled"}
Specifications
- KEDA Version:
docker.io/kedacore/keda:2.0.0 - Platform & Version: Linux, Azure Functions v3
- Kubernetes Version:
v.19.3 - Scaler(s):
azure-servicebus
Definitions
Scaled Object:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: attachment-thmbgn-funcs-scaledobject
spec:
scaleTargetRef:
name: attachment-thmbgn-funcs-scaled
triggers:
- metadata:
connectionFromEnv: AzureServiceBusPrimary
subscriptionName: images-function
topicName: attachments
type: azure-servicebus
- metadata:
connectionFromEnv: AzureServiceBusPrimary
subscriptionName: pdf-function
topicName: attachments
type: azure-servicebus
Deployment:
kind: Deployment
apiVersion: apps/v1
metadata:
name: attachment-thmbgn-funcs-scaled
labels:
env: pr-7095
spec:
replicas: 0
selector:
matchLabels:
app: attachment-thmbgn-funcs-scaled
template:
metadata:
labels:
app: attachment-thmbgn-funcs-scaled
env: pr-7095
spec:
containers:
- name: attachment-thmbgn-funcs-scaled
image: <censored>
ports:
- containerPort: 80
protocol: TCP
env:
- name: APPINSIGHTS_INSTRUMENTATIONKEY
valueFrom:
secretKeyRef:
name: attachment-thmbgn-funcs-secret
key: APPINSIGHTS_INSTRUMENTATIONKEY
- name: AttachmentsConnectionString
valueFrom:
secretKeyRef:
name: attachment-thmbgn-funcs-secret
key: AttachmentsConnectionString
- name: AzureServiceBusPrimary
valueFrom:
secretKeyRef:
name: attachment-thmbgn-funcs-secret
key: AzureServiceBusPrimary
- name: AzureStorageBlob
valueFrom:
secretKeyRef:
name: attachment-thmbgn-funcs-secret
key: AzureStorageBlob
- name: AzureWebJobsStorage
valueFrom:
secretKeyRef:
name: attachment-thmbgn-funcs-secret
key: AzureWebJobsStorage
- name: ASPNETCORE_ENVIRONMENT
value: pr-7095
- name: ASPNETCORE_FORWARDEDHEADERS_ENABLED
value: 'true'
- name: AzureFunctionsJobHost__functions__0
value: ImageThumbnailGenerator
- name: AzureFunctionsJobHost__functions__1
value: PdfThumbnailGenerator
- name: AttachmentImageHeight
value: '250'
- name: AttachmentImageWidth
value: '250'
- name: BlobContainer
value: attachments
- name: FUNCTIONS_EXTENSION_VERSION
value: ~3
- name: FUNCTIONS_WORKER_RUNTIME
value: dotnet
- name: ImageHeight
value: '250'
- name: ImageSubscription
value: images-function
- name: ImageWidth
value: '250'
- name: ImagesContainer
value: images
- name: PdfSubscription
value: pdf-function
- name: ThumbnailBlobContainer
value: thumbnails
- name: TopicName
value: attachments
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 25m
memory: 200Mi
livenessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 30
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
imagePullPolicy: IfNotPresent
restartPolicy: Always
strategy:
type: Recreate
Secret (censored of course):
kind: Secret
apiVersion: v1
metadata:
name: attachment-thmbgn-funcs-secret
data:
APPINSIGHTS_INSTRUMENTATIONKEY: <censored-base64-value>
AttachmentsConnectionString: >-
<censored-base64-value>
AzureServiceBusPrimary: >-
<censored-base64-value>
AzureStorageBlob: >-
<censored-base64-value>
AzureWebJobsStorage: >-
<censored-base64-value>
type: Opaque
If you need any more information from me please do let me know, but I hope this is everything you need. Please note I have modified the definitions/logs very slightly just to remove sensitive company information but it all matches up.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (18 by maintainers)
I’m afraid I’ll need @ahmelsayed for this
I don’t actually work at the company I was having these issues with anymore so I’d have to set a few things up but sure, happy to help. Simplest repo I know of is to try and use the Service Bus scaler using a key with only Send/Listen permissions (not Manage) which should be easy enough to try out.
Just checked the KEDA documentation and it says that you need a key with Manage permission so hopefully others won’t get caught up with this though, looks like it was in the docs when I was implementing this but apparently I can’t read 😄
@MattJeanes thanks for the investigation. We can keep this issue opened till https://github.com/Azure/azure-service-bus-go/issues/229 is resolved.
And if that issue won’t get fixed in foreseeable future, we can try to validate the connection string in KEDA.
Thanks for reporting, we’ll get to the bottom of this.
@ahmelsayed can you pick this up?