kubernetes: HPA V2 cannot get custom metrics info

Description of problem: HPA V2 cannot get custom metrics info.

Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----				-------------	--------	------		-------
  31m		6s		63	horizontal-pod-autoscaler			Warning		FailedGetObjectMetric		unable to get metric http_requests_total: Service on default sample-metrics-app/unable to fetch metrics from API: unable to map kind Service to resource: no matches for /, Kind=Service
  31m		6s		63	horizontal-pod-autoscaler			Warning		FailedComputeMetricsReplicas	failed to get object metric value: unable to get metric http_requests_total: Service on default sample-metrics-app/unable to fetch metrics from API: unable to map kind Service to resource: no matches for /, Kind=Service

kubernetes version: v1.7.0-alpha.3.129+20fa30e4b5a256

How reproducible: Always

Steps to Reproduce:

1. Enable kube DNS, start controller_manager with --horizontal-pod-autoscaler-use-rest-clients=“true” and start apiserver with --runtime-config=“api/all=true”

$ kubectl api-versions | grep autoscaling
autoscaling/v1
autoscaling/v2alpha1

1. Deploy the Prometheus Operator for monitoring Services in the cluster

kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/prometheus-operator.yaml
kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/sample-prometheus-instance.yaml

1. Deploy a custom metrics API Server and a sample app

kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/custom-metrics.yaml
kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/sample-metrics-app.yaml

1. Check hpa status.

# kubectl describe hpa sample-metrics-app-hpa

Actual results: hpa cannot get custom metrics, current value is “unknown” 4. Check hpa status.

# kubectl describe hpa sample-metrics-app-hpa
Name:							sample-metrics-app-hpa
Namespace:						default
Labels:							<none>
Annotations:						kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2alpha1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sample-metrics-app-hpa","namespace":"default...
CreationTimestamp:					Wed, 10 May 2017 04:29:50 -0400
Reference:						Deployment/sample-metrics-app
Metrics:						( current / target )
  "http_requests_total" on Service/sample-metrics-app:	<unknown> / 100
Min replicas:						2
Max replicas:						10
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----				-------------	--------	------		-------
  31m		6s		63	horizontal-pod-autoscaler			Warning		FailedGetObjectMetric		unable to get metric http_requests_total: Service on default sample-metrics-app/unable to fetch metrics from API: unable to map kind Service to resource: no matches for /, Kind=Service
  31m		6s		63	horizontal-pod-autoscaler			Warning		FailedComputeMetricsReplicas	failed to get object metric value: unable to get metric http_requests_total: Service on default sample-metrics-app/unable to fetch metrics from API: unable to map kind Service to resource: no matches for /, Kind=Service

Expected results: HPA V2 can get custom metrics info correctly.

Addition info:

# kubectl get all
NAME                                      READY     STATUS    RESTARTS   AGE
po/prometheus-operator-1303517395-dsnkk   1/1       Running   0          9m
po/prometheus-sample-metrics-prom-0       2/2       Running   0          9m
po/sample-metrics-app-2206349241-76xc4    1/1       Running   0          8m
po/sample-metrics-app-2206349241-gvvv7    1/1       Running   0          8m

NAME                      CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
svc/kubernetes            10.0.0.1     <none>        443/TCP          11m
svc/prometheus-operated   None         <none>        9090/TCP         9m
svc/sample-metrics-app    10.0.0.177   <none>        80/TCP           8m
svc/sample-metrics-prom   10.0.0.184   <nodes>       9090:30999/TCP   9m

NAME                                          DESIRED   CURRENT   AGE
statefulsets/prometheus-sample-metrics-prom   1         1         9m

NAME                         REFERENCE                       TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
hpa/sample-metrics-app-hpa   Deployment/sample-metrics-app   <unknown> / 100   2         10        2          8m

NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/prometheus-operator   1         1         1            1           9m
deploy/sample-metrics-app    2         2         2            2           8m

NAME                                DESIRED   CURRENT   READY     AGE
rs/prometheus-operator-1303517395   1         1         1         9m
rs/sample-metrics-app-2206349241    2         2         2         8m

# kubectl get hpa.v2alpha1.autoscaling sample-metrics-app-hpa -o yaml
apiVersion: autoscaling/v2alpha1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"autoscaling/v2alpha1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sample-metrics-app-hpa","namespace":"default"},"spec":{"maxReplicas":10,"metrics":[{"object":{"metricName":"http_requests_total","target":{"kind":"Service","name":"sample-metrics-app"},"targetValue":100},"type":"Object"}],"minReplicas":2,"scaleTargetRef":{"kind":"Deployment","name":"sample-metrics-app"}}}
  creationTimestamp: 2017-05-10T08:29:50Z
  name: sample-metrics-app-hpa
  namespace: default
  resourceVersion: "25672"
  selfLink: /apis/autoscaling/v2alpha1/namespaces/default/horizontalpodautoscalers/sample-metrics-app-hpa
  uid: d59e067f-355a-11e7-9933-0ef2c87c59d8
spec:
  maxReplicas: 10
  metrics:
  - object:
      metricName: http_requests_total
      target:
        kind: Service
        name: sample-metrics-app
      targetValue: "100"
    type: Object
  minReplicas: 2
  scaleTargetRef:
    kind: Deployment
    name: sample-metrics-app
status:
  currentMetrics: []
  currentReplicas: 2
  desiredReplicas: 0