kubernetes: Cannot update HPA objects using object metrics with AverageValue

Hitting this issue as well. There is a workaround tho

kubectl edit hpa.v2beta2.autoscaling <your-hpa>

Then spot every spec.metrics.*.object that uses AverageValue. Remove the .value ( that in my case is always "0").

After that, you can freely edit w/e you want from the rest of the document. The bad news, the .value: "0" returns after saving, so you need to delete it every time you want to edit again.

@k8s-triage-robot: Closing this issue.

I meet the same question in k8s 1.22 , and debug it

when create v2beta2 hpa

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: sample-app
  namespace: default
spec:
  maxReplicas: 10
  metrics:
  - object:
      describedObject:
        kind: Service
        name: sample-app
      metric:
        name: http_requests_count
      target:
        averageValue: "1"
        type: AverageValue
    type: Object
  minReplicas: 7
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app

and the object send to kube-apiserver , and v2beta2 hpa convert to internal hpa object. when read the internal hpa obj to v1 hpa obj, because metrics switch to annotation to the v1 hpa object, we can found the annotaion

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"False","lastTransitionTime":"2022-09-29T12:31:03Z","reason":"FailedGetScale","message":"the
      HPA controller was unable to get the target''s current scale: deployments/scale.apps
      \"sample-app\" not found"}]'
    autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Object","object":{"target":{"kind":"Service","name":"sample-app"},"metricName":"http_requests_count","targetValue":"0","averageValue":"1"}}]'
  creationTimestamp: "2022-09-29T12:30:48Z"
  name: sample-app
  namespace: default
  resourceVersion: "9314762"

in annotion exist “targetValue”:“0”

In v1 hpa define in vendor/k8s.io/api/autoscaling/v1/types.go:227, the TargetValue is resource.Quantity type.

type ObjectMetricSource struct {
	// target is the described Kubernetes object.
	Target CrossVersionObjectReference `json:"target" protobuf:"bytes,1,name=target"`

	// metricName is the name of the metric in question.
	MetricName string `json:"metricName" protobuf:"bytes,2,name=metricName"`
	// targetValue is the target value of the metric (as a quantity).
	TargetValue resource.Quantity `json:"targetValue" protobuf:"bytes,3,name=targetValue"``
---
}

and in v2beta2 hpa define in vendor/k8s.io/api/autoscaling/v2beta2/types.go:320, the TargetValue is *resource.Quantity type.

// MetricTarget defines the target value, average value, or average utilization of a specific metric
type MetricTarget struct {
	// type represents whether the metric type is Utilization, Value, or AverageValue
	Type MetricTargetType `json:"type" protobuf:"bytes,1,name=type"`
	// value is the target value of the metric (as a quantity).
	// +optional
	Value *resource.Quantity `json:"value,omitempty" protobuf:"bytes,2,opt,name=value"`
	// averageValue is the target value of the average of the
	// metric across all relevant pods (as a quantity)
	// +optional
	AverageValue *resource.Quantity `json:"averageValue,omitempty" protobuf:"bytes,3,opt,name=averageValue"`
	// averageUtilization is the target value of the average of the
	// resource metric across all relevant pods, represented as a percentage of
	// the requested value of the resource for the pods.
	// Currently only valid for Resource metric source type
	// +optional
	AverageUtilization *int32 `json:"averageUtilization,omitempty" protobuf:"bytes,4,opt,name=averageUtilization"`
}

the TargetValue is not * v1, and in v2beta2 hpa TargetValue is *.

then , the annotation of v1 hpa object will convert to internal hpa object, the “targetValue”:“0” in anntation be convert to internal hpa object, the target value is set to 0 not the nil

debug the problem in k8s 1.22.9 can fix the problem

diff --git a/pkg/apis/autoscaling/v1/conversion.go b/pkg/apis/autoscaling/v1/conversion.go
index bf94d48ae17..09fd8f9f337 100644
--- a/pkg/apis/autoscaling/v1/conversion.go
+++ b/pkg/apis/autoscaling/v1/conversion.go
@@ -90,9 +90,13 @@ func Convert_v1_ObjectMetricSource_To_autoscaling_ObjectMetricSource(in *autosca
 
        out.Target = autoscaling.MetricTarget{
                Type:         metricType,
-               Value:        &in.TargetValue,
                AverageValue: in.AverageValue,
        }
+
+       if in.TargetValue.Value() != 0 {
+               out.Target.Value = &in.TargetValue
+       }
+
        out.DescribedObject = autoscaling.CrossVersionObjectReference{
                Kind:       in.Target.Kind,
                Name:       in.Target.Name,

@egoroof: You can’t reopen an issue/PR unless you authored it or you are a collaborator.

Is there an update to this problem? We were using v2beta2 and switched to v2. We now get the same error we apply a helm deployment to the service that has an HPA with an Object resource.

We’ve deleted all the HPA’s manually and redeploying with helm works. Except its building the Object reference like this

  metrics:
  - object:
      describedObject:
        apiVersion: /v1
        kind: Namespace
        name: service-ui
      metric:
        name: service_requests_per_second
      target:
        averageValue: 500m
        type: AverageValue
        value: "0"

If we try to do a helm upgrade install after this to the same service, it fails with the same error.

The work around suggested by setting the value of value to 1 sounds like it should work and not interferre with our type of AverageValue. But why is value needed if we are not using the type of Value . This seems like a major oversight and redundant to include values if not in use if this is intended.

I using AverageValue target type, not Value - this is all about what this issue is. For AverageValue target type “averageValue” property used, “value” property is just ignored by AverageValue target type. My metric object look like this:

type: Object
object:
  metric:
    name: nginx_ingress_controller_requests_rate
  describedObject:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    name: customers
  target:
    type: AverageValue
    averageValue: "50"
    # Workaround for this bug https://github.com/kubernetes/kubernetes/issues/87733
    # remove it when bug will be fixed
    value: "1"

Your example above have different schema, it seems you are using different API version of HPA

Hitting this issue as well. There is a workaround tho

kubectl edit hpa.v2beta2.autoscaling <your-hpa>

Then spot every spec.metrics.*.object that uses AverageValue. Remove the .value ( that in my case is always "0").

After that, you can freely edit w/e you want from the rest of the document. The bad news, the .value: "0" returns after saving, so you need to delete it every time you want to edit again.

Unfortunatelly, this workaround not very aplicable for our workflow with automated HPA updates. So I just set spec.metrics[0].object.target.value property to “1” - and it seems works, satisfying that mistaken validation rule and being just ignored for AverageValue

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Encountering the same issue, it breaks our CI pipeline. Can we reopen this?

Encountering the same error, are there any updates?

An update: this doesn’t only affect the label command. Any attempt to patch the affected HPA object (i.e. changing minReplicas) will return the same error.