kubernetes: Missing validation for HPA annotation

Recently, I tried to create an HPA object with an empty value for the autoscaling.alpha.kubernetes.io/conditions annotation and even though the response from apiserver is a 500, the object itself still got created (I verified this by querying for the key from the etcd). And since then kubectl get/list hpa calls are failing because apiserver seems to be unable to parse that particular object. Even worse, I wasn’t even able to delete that object through the k8s api (had to delete the key directly from etcd).

This was on a v1.14.9 cluster.

Here’s a minimal hpa object to reproduce the issue (both v2beta1 and v2beta2 seem to have it):

{
    "kind": "HorizontalPodAutoscaler",
    "apiVersion": "autoscaling/v2beta1",
    "metadata": {
        "name": "bad-hpa-object",
        "annotations": {
            "autoscaling.alpha.kubernetes.io/conditions": ""
        }
    },
    "spec": {
        "scaleTargetRef": {
            "kind": "Deployment",
            "name": "test",
            "apiVersion": "apps/v1"
        },
        "maxReplicas": 1
    }
}

Failing “create hpa” request:

$ kubectl create -f bad-hpa-object.json --v=8
...
I0302 18:45:37.912801   25603 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unexpected end of JSON input","code":500}
I0302 18:45:37.913015   25603 helpers.go:196] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"hpa.json\": unexpected end of JSON input",
  "code": 500
}]

Kubectl call failing after above request:

$ kubectl get hpa
Error from server: unexpected end of JSON input

But object itself is present in etcd and I manually deleted it:

$ ETCDCTL_API=3 etcdctl get --keys-only --prefix "/registry/horizontalpodautoscalers/"
/registry/horizontalpodautoscalers/default/bad-hpa-object
$
$ etcdctl del /registry/horizontalpodautoscalers/default/bad-hpa-object
1

Which fixed the issue:

$ kubectl get hpa
No resources found.

cc @kubernetes/sig-api-machinery-bugs (I’m tagging apimachinery here because even though this could be a missing validation for a particular API owned by @kubernetes/sig-autoscaling-bugs , I’m wondering why apiserver created the object even though it returned a 500 to the client)

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 19 (19 by maintainers)

Most upvoted comments

Per @liggitt’s advice, I was looking into the conversion code today along with Harish Kuna. The bug seems to be coming from (ref):

if currentConditionsEnc, hasCurrentConditions := out.Annotations[autoscaling.HorizontalPodAutoscalerConditionsAnnotation]; hasCurrentConditions {
	var currentConditions []autoscalingv1.HorizontalPodAutoscalerCondition
	if err := json.Unmarshal([]byte(currentConditionsEnc), &currentConditions); err != nil {
		return err
	}

part of Convert_v1_HorizontalPodAutoscaler_To_autoscaling_HorizontalPodAutoscaler function. And the problem is that the annotation value being passed is an empty string and json.Unmarshal throws “unexpected end of JSON input” in such case.

So similar to what we do elsewhere, we should add an extra check in the if condition that currentConditionsEnc != "".

Btw - when I set the value of the annotation to empty array "[]" in the object, it gets created fine. So it’s almost certainly the bug above.

shyamjvs on Mar 11, 2020