keda: external.metrics API returns empty resource list when no scaledObject or scaledJob defined using external.metrics

Report

The Keda 2 external.metrics api returns an empty resources list, if no scalers using external.metrics are defined. The Go kubernetes client will error on an empty resource list https://github.com/kubernetes/client-go/blob/release-11.0/discovery/cached/memory/memcache.go#L215 .

In our specific case Flux CD is using the Go client and logs the following error if we have no ScaledObjects or ScaledJobs defined using external metrics:

ts=2021-03-25T03:21:02.663577928Z caller=main.go:276 type="internal kubernetes error" kubernetes_caller=k8s.io/client-go@v0.17.4/discovery/cached/memory/memcache.go:199 err="couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1"

We defined a dummy Prometheus scaledObject and the error log went away.

Confirmed an available apiservice

$ kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    fluxcd.io/sync-checksum: e97083f536ffb5c8f9b096037e7cde536936f128
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{"fluxcd.io/sync-checksum":"e97083f536ffb5c8f9b096037e7cde536936f128"},"labels":{"app.kubernetes.io/name":"v1beta1.external.metrics.k8s.io","app.kubernetes.io/part-of":"keda-operator","app.kubernetes.io/version":"2.2.0","environment":"zre_infra","fluxcd.io/sync-gc-mark":"sha256.QdyI5mN2VE3Gd1Xfdog2xdpfkEPERSAK6KUqCyKqwYI","owner":"zre"},"name":"v1beta1.external.metrics.k8s.io"},"spec":{"group":"external.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"keda-metrics-apiserver","namespace":"tools"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: "2021-04-12T22:03:30Z"
  labels:
    app.kubernetes.io/name: v1beta1.external.metrics.k8s.io
    app.kubernetes.io/part-of: keda-operator
    app.kubernetes.io/version: 2.2.0
    environment: zre_infra
    fluxcd.io/sync-gc-mark: sha256.QdyI5mN2VE3Gd1Xfdog2xdpfkEPERSAK6KUqCyKqwYI
    owner: zre
  name: v1beta1.external.metrics.k8s.io
  resourceVersion: "193189690"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io
  uid: d1f0c9d4-104e-4e3c-801a-565b2c539164
spec:
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: keda-metrics-apiserver
    namespace: tools
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2021-04-12T22:03:45Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

Then confirmed the empty resource list

# kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" -n tools
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[]}

Once dummy Prometheus scaled object was created, resource list was non empty

# kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" -n tools
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[{"name":"prometheus-http---prometheus-zymergen-net-9090-http_requests_total","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}]}

Expected Behavior

apiservice should not return an empty resource list for external.metrics as this will cause the K8s Go client to error.

Actual Behavior

Until a ScaledObject or ScaledJob is defined using external.metric the apiservice returns an empty resources list

Steps to Reproduce the Problem

1. Install keda2
2. Use the Go k8s client against the external.metrics.k8s.io/v1beta1 api provided by Keda apiservice before defining any scalers
3. Go client errors on empty resource list

Logs from KEDA operator

No response

KEDA Version

No response

Kubernetes Version

No response

Platform

No response

Scaler Details

No response

Anything else?

I originally asked about this in https://github.com/kedacore/keda/discussions/1698, and asked to file an issue. Finally circling back to file an actual issue.