keda: Connection error from keda-operator-metrics-apiserver logs

Report

After updating KEDA to version 2.9.2 (via chart version 2.9.3) from version 2.8.1 the metrics apiserver logs shows connection issue from the keda operator service at port 9666

Expected Behavior

The keda metrics apiserver to be able to use gRPC connection to successfully get metrics from operator

Actual Behavior

When I deployed the latest KEDA version 2.9.2 started seeing the below error in the metrics apiserver logs,

I0201 19:41:19.967541 1 main.go:162] keda_metrics_adapter “msg”=“Connecting Metrics Service gRPC client to the server” “address”=“keda-operator.mynamespace.svc.cluster.local:9666”

Err: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: "HTTP/1.1 502 Bad Gateway\r\n

Steps to Reproduce the Problem

Update KEDA version from 2.8.1 to 2.9.2

Logs from KEDA operator

I0201 19:41:19.967541       1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.<MYNAMESPACE>.svc.cluster.local:9666"

Err: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 502 Bad Gateway\\r\\n

KEDA Version

2.9.2

Kubernetes Version

1.23

Platform

Amazon Web Services

Scaler Details

No response

Anything else?

The keda service (13-keda-service.yaml) was introduced with the latest version referring to this issue https://github.com/kedacore/keda/issues/3920 as this service wasn’t there in the older version

I can try KEDA_USE_METRICS_SERVICE_GRPC=false but as it will be deprecated, wanted to see how to resolve the issue

KEDA runs behind enterprise proxy and the values are set via HTTP_PROXY HTTPS_PROXY and NO_PROXY