keda: couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

Report

Well, it happens all the time, I can’t use KEDA locally for testing.

The error :

E0210 10:57:59.544134  457383 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

Tried on:

  • Inside a VM
    • minikube
      • metrics-server as addon
      • metrics-server from helm
      • keda from helm
      • keda from apply
    • docker desktop
      • metrics-server from helm
      • keda from helm
      • keda from apply
    • microk8s
      • metrics-server as addon
      • metrics-server from helm
      • keda from helm
      • keda from apply
  • Bare metal
    • docker desktop
      • metrics-server from helm
      • keda from helm
      • keda from apply

Expected Behavior

Proper communication with metric-server and presenting the resources.

Actual Behavior

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[]}

I’ve made some stress tests and the deployments weren’t being scaled.

Steps to Reproduce the Problem

Simple example :

# start any kubectl context : minikube, docker-desktop, microk8s

# enable metrics-server
# or simply run this
helm upgrade --install metrics-server metrics-server/metrics-server -n kube-system --set args={--kubelet-insecure-tls}

# wait and make sure that HPA and "top nodes" are working

# deploy keda
# add its repo from website... then
kubectl create namespace keda
helm install keda kedacore/keda -n keda

After that, most of the times simply running kubectl get all -n keda is enough to raise the error. In a very few attempts I had to try creating a ScaledObject and then it happened immediately. I can’t even check if it is a problem with my ScaledObject because the error keeps occurring even after deleting the objects.

Logs from KEDA operator

microk8s kubectl get pods -n keda
E0210 17:12:43.288248 1396573 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-85f98c6655-bq2zn   1/1     Running   0          21m
keda-operator-97b74b8c8-xjchw             1/1     Running   0          21m


microk8s kubectl logs -n keda keda-operator-97b74b8c8-xjchw -c keda-operator
E0210 17:12:45.963719 1396717 memcache.go:255] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
2023-02-10T19:51:26Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-02-10T19:51:26Z    INFO    setup   Starting manager
2023-02-10T19:51:26Z    INFO    setup   KEDA Version: 2.9.2
2023-02-10T19:51:26Z    INFO    setup   Git Commit: 9bc3f66578a08cdfe084468ea3ef998fa6bf3bb0
2023-02-10T19:51:26Z    INFO    setup   Go Version: go1.18.8
2023-02-10T19:51:26Z    INFO    setup   Go OS/Arch: linux/amd64
2023-02-10T19:51:26Z    INFO    setup   Running on Kubernetes 1.26      {"version": "v1.26.1"}
I0210 19:51:26.316433       1 leaderelection.go:248] attempting to acquire leader lease keda/operator.keda.sh...
2023-02-10T19:51:26Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-02-10T19:51:26Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0210 19:51:26.354840       1 leaderelection.go:258] successfully acquired lease keda/operator.keda.sh
2023-02-10T19:51:26Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
2023-02-10T19:51:26Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2023-02-10T19:51:26Z    INFO    Starting Controller     {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2023-02-10T19:51:26Z    INFO    grpc_server     Starting Metrics Service gRPC Server    {"address": ":9666"}
2023-02-10T19:51:26Z    INFO    Starting EventSource    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2023-02-10T19:51:26Z    INFO    Starting Controller     {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2023-02-10T19:51:26Z    INFO    Starting EventSource    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
2023-02-10T19:51:26Z    INFO    Starting Controller     {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2023-02-10T19:51:26Z    INFO    Starting EventSource    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
2023-02-10T19:51:26Z    INFO    Starting Controller     {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2023-02-10T19:51:26Z    INFO    Starting workers        {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}
2023-02-10T19:51:26Z    INFO    Starting workers        {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2023-02-10T19:51:26Z    INFO    Starting workers        {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2023-02-10T19:51:26Z    INFO    Starting workers        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "worker count": 5}

KEDA Version

2.9.2

Kubernetes Version

1.26

Platform

None

Scaler Details

CPU, Memory

Anything else?

Yes, I have ALREADY seen the workaround comment on another issue telling to “create a dummy resource”.

I could be missing something, but as far as I can tell, the resources should be presented to me and if it is mandatory to do something else, it should be very explicit in the docs.

I don’t have any idea of what I should to to create this dummy resource and what exactly that means.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 18 (9 by maintainers)

Most upvoted comments

Hi, This is a problem introduced in the tooling by kube-client 0.26.0. We have already opened an issue to the upstream looking for a solution because KEDA (as other custom metrics server) follows the working way suggested but the custom-metrics server repo.

I know it’s annoying, but it isn’t something we can solve from our side without consolidating the solution with the upstream. The patch for this problem is to expose at least 1 metric from KEDA metrics server (CPU/Memory Scalers doesn’t expose any metric from KEDA’s metrics server).

How can you do it? You can create a ScaledObject for one workload, adding a trigger which does nothing, for example a cron trigger with desired replicas 1, this will expose that metric, patching the error you see. For example, you can do apply this in default namespace

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy-workload
spec:
  replicas: 0
  selector:
    matchLabels:
      app: dummy-workload
  template:
    metadata:
      labels:
        app: dummy-workload
    spec:
      containers:
      - name: dummy
        image: busybox
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: dummy-workload
spec:
  scaleTargetRef:
    name: dummy-workload
  minReplicaCount: 0
  maxReplicaCount: 1
  triggers:
  - type: cron
    metadata:
      timezone: Etc/UTC 
      start: 30 * * * *
      end: 45 * * * *
      desiredReplicas: "0"

This will create a dummy workload which will be always scale to 0, but exposing 1 metric and patching the issue

+17
JorTurFer on Feb 11, 2023

This commit, released as part of KEDA v2.11, patches the issue in KEDA side even if the tooling isn’t up-to-date

+1
JorTurFer on Jul 7, 2023
Read more comments on GitHub
← keda: Constants crashes in keda operator after deploying service controlled by scaledobject
keda: Datadog scaler is not able to find matching metrics →