digitalocean-cloud-controller-manager: Digital Ocean implementation of K8's does not allow metrics-server to function

Hello,

I raised an issue last night for this, but since I’ve had more time to sleep, wanted to raise it again and provide some more information.

I currently have a Digital Ocean Managed Kubernetes Cluster. I have some applications deployed and running on it.

I have configured Horizontal Pod Autoscaling for one of my deployments, but when running the kubectl get hpa command, I noticed the following in my output (<unknown> in the targets column):

NAME                              REFERENCE                                    TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
benjamin-maynard-io-fe            Deployment/benjamin-maynard-io-fe            <unknown>/80%   1         20        3          10h

I identified this was because I did not have either heapster or metrics-server running on my cluster. So went to install it as per the instructions on https://github.com/kubernetes-incubator/metrics-server

metrics-server successfully installs, and is running in the kube-system namespace:

NAME                                READY   STATUS    RESTARTS   AGE
csi-do-controller-0                 3/3     Running   0          42h
csi-do-node-dbvg5                   2/2     Running   0          42h
csi-do-node-lq97x                   2/2     Running   1          42h
csi-do-node-mvnrw                   2/2     Running   0          42h
kube-dns-55cf9576c4-4r466           3/3     Running   0          42h
kube-proxy-upbeat-lichterman-3mz4   1/1     Running   0          42h
kube-proxy-upbeat-lichterman-3mzh   1/1     Running   0          42h
kube-proxy-upbeat-lichterman-3mzi   1/1     Running   0          42h
metrics-server-7fbd9b8589-64x86     1/1     Running   0          9m48s

However, I am still getting no metrics.

Running kubectl get apiservice v1beta1.metrics.k8s.io -o yaml reveals:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: 2018-11-27T08:32:26Z
  name: v1beta1.metrics.k8s.io
  resourceVersion: "396557"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: f88f3576-f21e-11e8-8aed-fab39051e242
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2018-11-27T08:32:26Z
    message: 'no response from https://10.245.219.253:443: Get https://10.245.219.253:443:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

With the latter part of the output being of interest message: 'no response from https://10.245.219.253:443: Get https://10.245.219.253:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)'

I believe the above error message means that the kube-apiserver cannot speak to the metrics-server service. I believe this is due to the specifics of how the Digital Ocean Kubernetes master works.

I’ve performed some other general validation:

Service is configured:

Benjamins-MacBook-Pro:metrics-server benmaynard$ kubectl get service --namespace=kube-system
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kube-dns         ClusterIP   10.245.0.10      <none>        53/UDP,53/TCP   2d19h
metrics-server   ClusterIP   10.245.219.253   <none>        443/TCP         14m

metrics-server is up and running:

Benjamins-MacBook-Pro:metrics-server benmaynard$ kubectl logs metrics-server-7fbd9b8589-64x86 --namespace=kube-system
I1127 08:32:30.665197       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/11/27 08:32:32 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/11/27 08:32:32 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1127 08:32:32.981732       1 serve.go:96] Serving securely on [::]:443

Another customer has reported similar things: https://www.digitalocean.com/community/questions/cannot-get-kubernetes-horizonal-pod-autoscaler-or-metrics-server-working