kube-metrics-adapter: currentMetrics is null for external metrics hpa
Expected Behavior
Works well as my another hpa, in same environment.
$ k get hpa -n test -oyaml
apiVersion: v1
items:
- apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
annotations:
metric-config.external.istio-requests-total.prometheus/prometheus-server: http://prometheus-server.infra.svc
metric-config.external.istio-requests-total.prometheus/query: |
sum(
rate(
istio_requests_total{
destination_workload="podinfo",
destination_workload_namespace="test",
reporter="destination"
}[2m]
)
) /
count(
count(
container_memory_usage_bytes{
namespace="test",
pod=~"podinfo.*"
}
) by (pod)
)
creationTimestamp: "2024-06-06T11:06:40Z"
name: podinfo
namespace: test
resourceVersion: "68922388"
uid: 440d0a48-1e20-4eb3-b802-7a5df5274481
spec:
maxReplicas: 10
metrics:
- external:
metric:
name: istio-requests-total
selector:
matchLabels:
type: prometheus
target:
averageValue: "10"
type: AverageValue
type: External
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
status:
conditions:
- lastTransitionTime: "2024-06-06T11:06:48Z"
message: recommended size matches current size
reason: ReadyForNewScale
status: "True"
type: AbleToScale
- lastTransitionTime: "2024-06-06T11:29:33Z"
message: 'the HPA was able to successfully calculate a replica count from external
metric istio-requests-total(&LabelSelector{MatchLabels:map[string]string{type:
prometheus,},MatchExpressions:[]LabelSelectorRequirement{},})'
reason: ValidMetricFound
status: "True"
type: ScalingActive
- lastTransitionTime: "2024-06-06T12:37:07Z"
message: the desired replica count is less than the minimum replica count
reason: TooFewReplicas
status: "True"
type: ScalingLimited
currentMetrics:
- external:
current:
averageValue: "0"
metric:
name: istio-requests-total
selector:
matchLabels:
type: prometheus
type: External
currentReplicas: 1
desiredReplicas: 1
lastScaleTime: "2024-06-06T12:29:37Z"
kind: List
metadata:
resourceVersion: ""
Actual Behavior
currentMetrics is null, hpa not work.
$ k get hpa -n zone-dev aiservice -oyaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
annotations:
metric-config.external.dcgm-fi-dev-gpu-util.prometheus/prometheus-server: http://prometheus-server.infra.svc
metric-config.external.dcgm-fi-dev-gpu-util.prometheus/query: |
avg(
avg_over_time(
DCGM_FI_DEV_GPU_UTIL{
app="nvidia-dcgm-exporter",
container="service",
exported_namespace="zone-dev",
pod=~"aiservice-.*",
service="nvidia-dcgm-exporter"
}[1m]
)
)
creationTimestamp: "2024-06-06T12:37:33Z"
name: aiservice
namespace: zone-dev
resourceVersion: "68926327"
uid: f0e5f9cf-cc9e-4f60-b97f-0ad8a0727cfd
spec:
maxReplicas: 5
metrics:
- external:
metric:
name: dcgm-fi-dev-gpu-util
selector:
matchLabels:
type: prometheus
target:
averageValue: "50"
type: AverageValue
type: External
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: aiservice
status:
conditions:
- lastTransitionTime: "2024-06-06T12:37:45Z"
message: the HPA controller was able to get the target's current scale
reason: SucceededGetScale
status: "True"
type: AbleToScale
- lastTransitionTime: "2024-06-06T12:37:45Z"
message: scaling is disabled since the replica count of the target is zero
reason: ScalingDisabled
status: "False"
type: ScalingActive
currentMetrics: null
desiredReplicas: 0
Steps to Reproduce the Problem
- Install kube-metrics-adapter, with
cd .\docs; kubectl apply -f . - Create hpa for podinfo, with external metrics “istio-requests-total”, it works (details as upper)
- Create hpa for aiservice, with external metrics “cgm-fi-dev-gpu-util”, the hpa does not work, currentMetrics is null (detail configuration is as upper)
Here are logs, seems already got the metrics
$ k logs -n ks kube-metrics-adapter-7fd88d677c-fgr4t -f | grep cgm-fi-dev-gpu-util
time="2024-06-06T11:37:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:38:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (64) [type=prometheus]" provider=hpa
time="2024-06-06T11:39:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:40:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:41:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (0) [type=prometheus]" provider=hpa
time="2024-06-06T11:42:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (59) [type=prometheus]" provider=hpa
time="2024-06-06T11:43:29Z" level=info msg="Collected new external metric 'zone-dev/dcgm-fi-dev-gpu-util' (58) [type=prometheus]" provider=hpa
Here is prometheus, you can find the metrics is showed.
Specifications
- Version: 0.2.2
- Platform: k8s 1.29
- Subsystem:
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (17 by maintainers)
Let me close this ticket. Thanks for your excellent support. @szuecs @mikkeloscar
@johnzheng1975 I wanted to see the events, you shared
get hpaoutput, I want to seedescribe hpa.Thanks for your reminder.
Since the averageUtilization: 95 is based on request memory, it will not OOM if limit memory higher than request memory, am I right? Thanks.