keda: (2.7.1) ScaledObject creates HPA which cannot fetch metrics from external metrics api on GKE.
Report
When creating a ScaledObject using any scaler (tested on both cron and rabbitmq) scaling falls back to the fallback configuration and the following error is seen in the HPA:
the HPA was unable to compute the replica count: unable to get external metric dev/s0-cron-America-New_York-09xxx-017xxx/&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: bilayer-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-cron-America-New_York-09xxx-017xxx.external.metrics.k8s.io)
ScaledObject yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: bilayer-scaledobject
spec:
scaleTargetRef:
name: bilayer-worker # Mandatory. Must be in the same namespace as the ScaledObject
pollingInterval: 30 # Optional. Default: 30 seconds
cooldownPeriod: 300 # Optional. Default: 300 seconds
idleReplicaCount: 0 # Optional. Default: ignored, must be less than minReplicaCount
minReplicaCount: 1 # Optional. Default: 0
maxReplicaCount: 10 # Optional. Default: 100
fallback: # Optional. Section to specify fallback options
failureThreshold: 3 # Mandatory if fallback section is included
replicas: 1 # Mandatory if fallback section is included
advanced: # Optional. Section to specify advanced options
restoreToOriginalReplicaCount: true # Optional. Default: false
horizontalPodAutoscalerConfig: # Optional. Section to specify HPA related options
behavior: # Optional. Use to modify HPA's scaling behavior
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 100
periodSeconds: 15
triggers:
# - type: rabbitmq
# authenticationRef:
# name: rabbitmq-triggerauthentication
# kind: TriggerAuthentication
# metadata:
# protocol: amqp # Optional. Specifies protocol to use, either amqp or http, or auto to autodetect based on the `host` value. Default value is auto.
# mode: QueueLength # QueueLength or MessageRate
# value: "10" # message backlog or publish/sec. target per instance
# queueName: bilayer
- type: cron
metadata:
# Required
timezone: America/New_York # The acceptable values would be a value from the IANA Time Zone Database.
start: 0 9 * * *
end: 0 17 * * *
desiredReplicas: "1"
Expected Behavior
ScaledObject creates an HPA that properly reads scaling metrics.
Actual Behavior
ScaledObject seems to fall back to the fallback configuration.
HPA reports:
the HPA was unable to compute the replica count: unable to get external metric dev/s0-cron-America-New_York-09xxx-017xxx/&LabelSelector{MatchLabels:map[string[]string{scaledobject.keda.sh/name: bilayer-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get s0-cron-America-New_York-09xxx-017xxx.external.metrics.k8s.io)
Steps to Reproduce the Problem
- Create Cluster on GKE (not sure if this is important so including this step)
- Apply above (or similar) YAML
- Monitor HPA for metric errors.
Logs from KEDA operator
1.6528022296032584e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528022297567873e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528022297968023e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.652802244446022e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528022445112984e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528022445445988e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.65280224458145e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.6528022446254056e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.652802244787323e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528022448254182e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528022590182226e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528022594759974e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528022595400887e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528022598032768e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528022744933136e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528022746613815e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528022748660052e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528022894951754e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.652802289584336e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528022896254184e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.652802304075761e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.65280230462723e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528023048472593e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528023190977757e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.652802319549717e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.652802319907837e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528023345640056e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528023345830283e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528023346907315e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.6528023348869414e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528023491420298e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528023496362195e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528023497289135e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.652802349958667e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528023646527598e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528023647150707e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.652802364748541e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.6528023647979023e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528023649329114e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528023796383085e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528023797530763e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528023799494808e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528023942269444e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528023947710814e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528023948420932e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.652802395010182e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528024092641726e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.652802409861488e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528024242834988e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528024246880057e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528024247147412e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.652802424813949e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528024248409376e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.6528024248878617e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528024397365115e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528024397682452e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528024400065858e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528024547373724e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528024548552601e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528024550221486e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528024693378344e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528024697743478e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528024698021839e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.652802470125436e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.652802484356023e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528024847980127e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528024848200953e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528024849262798e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.652802485057372e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528024851525884e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528024998329105e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528024999585311e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528024999995399e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.65280250002823e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.652802500075199e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.652802500169612e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528025144090395e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528025149987106e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528025150665905e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528025294279487e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528025300182304e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528025300468438e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.652802530078954e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.6528025300992908e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.652802530209765e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528025444699776e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "superres-scaledobject", "namespace": "dev"}
1.6528025449360862e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.652802544967055e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528025450465286e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.6528025450797384e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528025451393692e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528025452273915e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528025599672227e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528025599841208e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "openpose-scaledobject", "namespace": "dev"}
1.6528025600982623e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528025601559558e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
1.6528025602552865e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "densepose-scaledobject", "namespace": "dev"}
1.6528025749848387e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.6528025750931385e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "tryon-scaledobject", "namespace": "dev"}
1.652802575178688e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "bilayer-scaledobject", "namespace": "dev"}
1.6528025900197892e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "modnet-scaledobject", "namespace": "dev"}
1.652802590142411e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "expose-scaledobject", "namespace": "dev"}
1.6528025901706774e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "hair-reposer-scaledobject", "namespace": "dev"}
1.6528025902378626e+09 INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "smplx-scaledobject", "namespace": "dev"}
KEDA Version
2.7.1
Kubernetes Version
1.23
Platform
Google Cloud
Scaler Details
Cron and RabbbitMQ
Anything else?
Keda Metrics API Server
I0517 15:26:42.418115 1 main.go:147] keda_metrics_adapter "msg"="KEDA Version: 2.7.1"
I0517 15:26:42.418188 1 main.go:148] keda_metrics_adapter "msg"="KEDA Commit: "
I0517 15:26:42.418196 1 main.go:149] keda_metrics_adapter "msg"="Go Version: go1.17.9"
I0517 15:26:42.418202 1 main.go:150] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64"
I0517 15:26:43.470873 1 request.go:665] Waited for 1.038036878s due to client-side throttling, not priority and fairness, request: GET:https://10.16.0.1:443/apis/node.k8s.io/v1?timeout=32s
I0517 15:26:46.027346 1 logr.go:261] keda_metrics_adapter/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"=":8080"
I0517 15:26:46.027766 1 provider.go:65] keda_metrics_adapter/provider "msg"="starting"
I0517 15:26:46.027788 1 main.go:214] keda_metrics_adapter "msg"="starting adapter..."
2022/05/17 15:26:46 Starting metrics server at :9022
I0517 15:26:46.028543 1 internal.go:362] keda_metrics_adapter "msg"="Starting server" "addr"={"IP":"::","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
I0517 15:26:46.028649 1 controller.go:178] keda_metrics_adapter/controller/scaledobject "msg"="Starting EventSource" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0517 15:26:46.028671 1 controller.go:178] keda_metrics_adapter/controller/scaledobject "msg"="Starting EventSource" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "source"="kind source: *v1alpha1.ScaledObject"
I0517 15:26:46.028687 1 controller.go:186] keda_metrics_adapter/controller/scaledobject "msg"="Starting Controller" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject"
I0517 15:26:46.129430 1 controller.go:220] keda_metrics_adapter/controller/scaledobject "msg"="Starting workers" "reconciler group"="keda.sh" "reconciler kind"="ScaledObject" "worker count"=1
I0517 15:26:46.607019 1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0517 15:26:47.081138 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0517 15:26:47.081277 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0517 15:26:47.081184 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0517 15:26:47.081383 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0517 15:26:47.081223 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0517 15:26:47.081519 1 secure_serving.go:266] Serving securely on :6443
I0517 15:26:47.081538 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0517 15:26:47.081555 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0517 15:26:47.081551 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0517 15:26:47.181673 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0517 15:26:47.181681 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0517 15:26:47.181940 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
Kubernetes Metric Server
metrics-server I0517 15:36:31.704448 1 serving.go:341] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
metrics-server I0517 15:36:39.811279 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
metrics-server I0517 15:36:39.811298 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
metrics-server I0517 15:36:39.811329 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
metrics-server I0517 15:36:39.811318 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
metrics-server I0517 15:36:39.811430 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
metrics-server I0517 15:36:39.811450 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
metrics-server I0517 15:36:39.899552 1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
metrics-server I0517 15:36:40.100012 1 secure_serving.go:202] Serving securely on :10250
metrics-server I0517 15:36:40.100421 1 tlsconfig.go:240] Starting DynamicServingCertificateController
metrics-server I0517 15:36:40.199481 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
metrics-server I0517 15:36:40.199530 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
metrics-server-nanny ERROR: logging before flag.Parse: I0517 15:36:25.403164 1 pod_nanny.go:68] Invoked by [/pod_nanny --config-dir=/etc/config --cpu=40m --extra-cpu=0.5m --memory=35Mi --extra-memory=4Mi --threshold=5 --deployment=metrics-server-v0.5.2 --container=metrics-server --poll-period=30000 --estimator=exponential --scale-down-delay=24h --minClusterSize=5 --use-metrics=true]
metrics-server-nanny ERROR: logging before flag.Parse: I0517 15:36:25.403271 1 pod_nanny.go:69] Version: 1.8.14
metrics-server-nanny ERROR: logging before flag.Parse: I0517 15:36:25.403294 1 pod_nanny.go:85] Watching namespace: kube-system, pod: metrics-server-v0.5.2-d56b865b7-tf8w5, container: metrics-server.
metrics-server-nanny ERROR: logging before flag.Parse: I0517 15:36:25.403310 1 pod_nanny.go:86] storage: MISSING, extra_storage: 0Gi
metrics-server I0517 15:36:40.199811 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
metrics-server-nanny ERROR: logging before flag.Parse: I0517 15:36:25.405315 1 pod_nanny.go:116] cpu: 40m, extra_cpu: 0.5m, memory: 35Mi, extra_memory: 4Mi
metrics-server-nanny ERROR: logging before flag.Parse: I0517 15:36:25.405358 1 pod_nanny.go:145] Resources: [{Base:{i:{value:40 scale:-3} d:{Dec:<nil>} s:40m Format:DecimalSI} ExtraPerNode:{i:{value:5 scale:-4} d:{Dec:<nil>} s: Format:DecimalSI} Name:cpu} {Base:{i:{value:36700160 scale:0} d:{Dec:<nil>} s:35Mi Format:BinarySI} ExtraPerNode:{i:{value:4194304 scale:0} d:{Dec:<nil>} s:4Mi Format:BinarySI} Name:memory}]
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (4 by maintainers)
Yes in the installation. I’ll provide a pr.