kubernetes: HPA not able to find CPU, looking at unrelated pod

What happened: I created HPAs for several Deployments using the helm yaml below:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata: 
  name: {{template "fullname" . }}
  namespace: default
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1
    kind: Deployment
    name: {{template "fullname" . }}
  maxReplicas: 5
  minReplicas: 1
  targetCPUUtilizationPercentage: 100

It works fine except for on my pod “metric” that has 3 containers in it. All containers have kubernetes resource requests and limits for CPU and memory. The problem is the HPA reports it cannot find the CPU usage.

kubectl get -o json hpa release-metric
{
    "apiVersion": "autoscaling/v1",
    "kind": "HorizontalPodAutoscaler",
    "metadata": {
        "annotations": {
            "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"True\",\"lastTransitionTime\":\"2018-05-09T15:17:51Z\",\"reason\":\"SucceededGetScale\",\"message\":\"the HPA controller was able to get the target's current scale\"},{\"type\":\"ScalingActive\",\"status\":\"False\",\"lastTransitionTime\":\"2018-05-09T15:17:51Z\",\"reason\":\"FailedGetResourceMetric\",\"message\":\"the HPA was unable to compute the replica count: missing request for cpu on container release-event in pod default/release-event-5486b6976c-q2btv\"}]"
        },
        "creationTimestamp": "2018-05-09T15:17:21Z",
        "name": "release-metric",
        "namespace": "default",
        "resourceVersion": "16249",
        "selfLink": "/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/release-metric",
        "uid": "11bee95f-539c-11e8-a15c-005056b43c41"
    },
    "spec": {
        "maxReplicas": 5,
        "minReplicas": 1,
        "scaleTargetRef": {
            "apiVersion": "extensions/v1beta1",
            "kind": "Deployment",
            "name": "release-metric"
        },
        "targetCPUUtilizationPercentage": 100
    },
    "status": {
        "currentReplicas": 1,
        "desiredReplicas": 0
    }
}

The odd thing is the “event” pod is completely unrelated to the “metric” pod. In another environment, it’s looking at another different pod “applications” that again has nothing to do with the Deployment “metric”. It seems to be getting confused on what pod to look for when there are multiple containers in the Deployment.
I created another HPA against another pod with 2 containers, and again the HPA is looking for CPU for unrelated pods.

What you expected to happen: CPU is correctly monitored for the pod and horizontally scaled.

How to reproduce it (as minimally and precisely as possible): Create a “deployment” HPA for a pod with 2 or more containers.

Anything else we need to know?: I tried creating an HPA against the ReplicaSet for my “metric” service and this works okay. The problem is, I’m not sure how to get the “ReplicaSet” name into the helm yaml to replace name: {{template "fullname" . }} since in my helm charts it’s turning into <release>-metric-<hash>.

Environment:

• Kubernetes version (use kubectl version): v1.9.1+icp
• Cloud provider or hardware configuration: vmware
• OS (e.g. from /etc/os-release): rhel 7.4
• Kernel (e.g. uname -a): 3.10.0-693.21.1.el7.x86_64

/kind bug /sig autoscaling /sig scalability