keda: Azure Monitor scaler doesn't work with Azure AD Workload Identity
Report
KEDA Version: 2.9.3
Git Commit: 0d6ce53cc7e03d82c589c0a7e1b16f82740930c
Go Version: go1.18.8
Go OS/Arch: linux/amd64
Kubernetes 1.24 {“version”: “v1.24.9”}
Not able to use Azure AD Workload Identity for authentication. Tested both with Azure AD Application and user-assigned managed identity.
Getting the following error:
2023-02-09T13:18:10Z ERROR azure_monitor_scaler error getting azure monitor metric {"type": "ScaledObject", "namespace": "default", "name": "aks-helloworld-one-scaledobject", "error": "error getting azure monitor metric CurrentConnections: insights.MetricsClient#List: Failure sending request: StatusCode=0 -- Original Error: context canceled"}
github.com/kedacore/keda/v2/pkg/scalers.(*azureMonitorScaler).GetMetricsAndActivity │
│ /workspace/pkg/scalers/azure_monitor_scaler.go:235
│ github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsForScaler
│ /workspace/pkg/scaling/cache/scalers_cache.go:88
│ github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
│ /workspace/pkg/scaling/scale_handler.go:443
│ github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
│ /workspace/pkg/metricsservice/server.go:45
│ github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
│ /workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
│ google.golang.org/grpc.(*Server).processUnaryRPC
│ /workspace/vendor/google.golang.org/grpc/server.go:1340
│ google.golang.org/grpc.(*Server).handleStream
│ /workspace/vendor/google.golang.org/grpc/server.go:1713
│ google.golang.org/grpc.(*Server).serveStreams.func1.2
│ /workspace/vendor/google.golang.org/grpc/server.go:965
The ScaledObject works when credential based authentication is being used.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: azure-monitor-trigger-auth
spec:
podIdentity:
provider: azure-workload
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: helloworld-one-scaledobject
spec:
scaleTargetRef:
kind: Deployment
name: helloworld-one
minReplicaCount: 1
maxReplicaCount: 3
triggers:
- type: azure-monitor
metadata:
resourceURI: Microsoft.Network/applicationgateways/myAppGTW
tenantId: xxxx
subscriptionId: xxxx
resourceGroupName: rg-metrics
metricName: CurrentConnections
metricAggregationInterval: "0:1:0"
metricAggregationType: Total
targetValue: "10"
authenticationRef:
name: azure-monitor-trigger-auth
Expected Behavior
Get metrics from Azure Monitor
Actual Behavior
Getting an error saying: Failure sending request: StatusCode=0 – Original Error: context canceled
Steps to Reproduce the Problem
- Create a ScaledObject using podIdentity: provider: azure-workload for Auth
Logs from KEDA operator
2023-02-09T13:18:10Z ERROR azure_monitor_scaler error getting azure monitor metric {"type": "ScaledObject", "namespace": "default", "name": "aks-helloworld-one-scaledobject", "error": "error getting azure monitor metric CurrentConnections: insights.MetricsClient#List: Failure sending request: StatusCode=0 -- Original Error: context canceled"}
github.com/kedacore/keda/v2/pkg/scalers.(*azureMonitorScaler).GetMetricsAndActivity │
│ /workspace/pkg/scalers/azure_monitor_scaler.go:235
│ github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsForScaler
│ /workspace/pkg/scaling/cache/scalers_cache.go:88
│ github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
│ /workspace/pkg/scaling/scale_handler.go:443
│ github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
│ /workspace/pkg/metricsservice/server.go:45
│ github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
│ /workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
│ google.golang.org/grpc.(*Server).processUnaryRPC
│ /workspace/vendor/google.golang.org/grpc/server.go:1340
│ google.golang.org/grpc.(*Server).handleStream
│ /workspace/vendor/google.golang.org/grpc/server.go:1713
│ google.golang.org/grpc.(*Server).serveStreams.func1.2
│ /workspace/vendor/google.golang.org/grpc/server.go:965
KEDA Version
2.9.3
Kubernetes Version
1.24
Platform
Microsoft Azure
Scaler Details
Azure Monitor
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 19 (14 by maintainers)
Don’t worry, just use the chained credential which uses the new identity SDK and trust in the implementation. We are already using it for Azure Service Bus scaler and it works 😄
If it uses the new SDK, you should use our chained credential because it handles the different pod identities and also adds az-cli credentials when needed.
I think this is handled by the SDK on top of
GetTokenfunction, I mean, providers don’t need to cache the token as the token “is the same” for any credential provider, it’s cached at caller level and not as part of the provider (but I need to review SDKs to be 100% sure)BTW @v-shenoy , IDK if you will change to the Azure-sdk-for-golang (because IDK if it supports azure monitor), but if you change to it, there is a chained token credential that I created for the ServiceBus scaler to register az-cli and (msi/wi) for local debugging. Please use it instead of just adding the specific token credential, that will help with local debugging of the scaler with podIdentities. Last thing about this, last week we merged the e2e tests for this scaler, with and without pod identities, so I hope you can be more safe with your changes 😃
You are right, there are some changes and I forgot to document them properly but as you are not the first one who noticed this, the PR is already open 😄
You can check in the PR the extra steps you need to do for running KEDA locally (which basically are to generate the certificates using openssl in the correct folder)