kube-metrics-adapter: currentMetrics is null for external metrics hpa

Expected Behavior

Works well as my another hpa, in same environment. image

$ k get hpa -n test -oyaml
apiVersion: v1
items:
- apiVersion: autoscaling/v2
  kind: HorizontalPodAutoscaler
  metadata:
    annotations:
      metric-config.external.istio-requests-total.prometheus/prometheus-server: http://prometheus-server.infra.svc
      metric-config.external.istio-requests-total.prometheus/query: |
        sum(
            rate(
                istio_requests_total{
                  destination_workload="podinfo",
                  destination_workload_namespace="test",
                  reporter="destination"
                }[2m]
            )
        ) /
        count(
          count(
            container_memory_usage_bytes{
              namespace="test",
            pod=~"podinfo.*"
            }
          ) by (pod)
        )
    creationTimestamp: "2024-06-06T11:06:40Z"
    name: podinfo
    namespace: test
    resourceVersion: "68922388"
    uid: 440d0a48-1e20-4eb3-b802-7a5df5274481
  spec:
    maxReplicas: 10
    metrics:
    - external:
        metric:
          name: istio-requests-total
          selector:
            matchLabels:
              type: prometheus
        target:
          averageValue: "10"
          type: AverageValue
      type: External
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: podinfo
  status:
    conditions:
    - lastTransitionTime: "2024-06-06T11:06:48Z"
      message: recommended size matches current size
      reason: ReadyForNewScale
      status: "True"
      type: AbleToScale
    - lastTransitionTime: "2024-06-06T11:29:33Z"
      message: 'the HPA was able to successfully calculate a replica count from external
        metric istio-requests-total(&LabelSelector{MatchLabels:map[string]string{type:
        prometheus,},MatchExpressions:[]LabelSelectorRequirement{},})'
      reason: ValidMetricFound
      status: "True"
      type: ScalingActive
    - lastTransitionTime: "2024-06-06T12:37:07Z"
      message: the desired replica count is less than the minimum replica count
      reason: TooFewReplicas
      status: "True"
      type: ScalingLimited
    currentMetrics:
    - external:
        current:
          averageValue: "0"
        metric:
          name: istio-requests-total
          selector:
            matchLabels:
              type: prometheus
      type: External
    currentReplicas: 1
    desiredReplicas: 1
    lastScaleTime: "2024-06-06T12:29:37Z"
kind: List
metadata:
  resourceVersion: ""

Actual Behavior

currentMetrics is null, hpa not work. image

$ k get hpa -n zone-dev aiservice  -oyaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    metric-config.external.dcgm-fi-dev-gpu-util.prometheus/prometheus-server: http://prometheus-server.infra.svc
    metric-config.external.dcgm-fi-dev-gpu-util.prometheus/query: |
      avg(
        avg_over_time(
          DCGM_FI_DEV_GPU_UTIL{
            app="nvidia-dcgm-exporter",
            container="service",
            exported_namespace="zone-dev",
            pod=~"aiservice-.*",
            service="nvidia-dcgm-exporter"
          }[1m]
        )
      )
  creationTimestamp: "2024-06-06T12:37:33Z"
  name: aiservice
  namespace: zone-dev
  resourceVersion: "68926327"
  uid: f0e5f9cf-cc9e-4f60-b97f-0ad8a0727cfd
spec:
  maxReplicas: 5
  metrics:
  - external:
      metric:
        name: dcgm-fi-dev-gpu-util
        selector:
          matchLabels:
            type: prometheus
      target:
        averageValue: "50"
        type: AverageValue
    type: External
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: aiservice
status:
  conditions:
  - lastTransitionTime: "2024-06-06T12:37:45Z"
    message: the HPA controller was able to get the target's current scale
    reason: SucceededGetScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2024-06-06T12:37:45Z"
    message: scaling is disabled since the replica count of the target is zero
    reason: ScalingDisabled
    status: "False"
    type: ScalingActive
  currentMetrics: null
  desiredReplicas: 0

Steps to Reproduce the Problem

  1. Install kube-metrics-adapter, with cd .\docs; kubectl apply -f .
  2. Create hpa for podinfo, with external metrics “istio-requests-total”, it works (details as upper)
  3. Create hpa for aiservice, with external metrics “cgm-fi-dev-gpu-util”, the hpa does not work, currentMetrics is null (detail configuration is as upper)

Here are logs, seems already got the metrics

$ k logs -n ks kube-metrics-adapter-7fd88d677c-fgr4t  -f | grep cgm-fi-dev-gpu-util

time="2024-06-06T11:37:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:38:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (64) [type=prometheus]" provider=hpa
time="2024-06-06T11:39:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:40:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:41:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:42:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (59) [type=prometheus]" provider=hpa
time="2024-06-06T11:43:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (58) [type=prometheus]" provider=hpa

Here is prometheus, you can find the metrics is showed.

image

image

Specifications

  • Version: 0.2.2
  • Platform: k8s 1.29
  • Subsystem:

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Most upvoted comments

Let me close this ticket. Thanks for your excellent support. @szuecs @mikkeloscar

@johnzheng1975 I wanted to see the events, you shared get hpa output, I want to see describe hpa.

Thanks for your reminder.

Unrelated to the issue: One other small thing that you likely want to change is memory averageUtilization: 95 would mean that it scales-out only at +10%, which is 105%, which is likely already OOM.

Since the averageUtilization: 95 is based on request memory, it will not OOM if limit memory higher than request memory, am I right? Thanks.