hcloud-cloud-controller-manager: Problems with Rancher deployed clusters

Hi! I am testing this controller with Rancher clusters, and for some reason the metrics server nor the Prometheus/Grafana monitoring installed by Rancher seem to work. kubectl top nodes returns error: metrics not available yet even after waiting for a while, and the monitoring API is never coming up.

I can’t remember the details, but I did test this a little months ago and had similar problems because of the IP addresses. Before installing the controller, kubectl get nodes -owide was showing the IP addresses as internal, while after installing the controller they are shown as external and there is no internal IP. I can’t remember how I found out there was a link between this change and the metrics servert/API not being available. Am I missing something? I made sure the kubelet is configured with cloud-provider = external.

Thanks!

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 19 (6 by maintainers)

Most upvoted comments

@vitobotta if you find any solution for 1) it would be very nice. We are in a early testing stage with R2 and hetzner cloud at the moment and are hanging here too, since it is not a suitable solution for us to keep clusters in same project. In our case some clusters are not even the same hetzner customers.

From a security perspective, i would recommend separating every customer, so one customer could bring your clusters down.

LKaemmerling on Jul 23, 2020

Hi @LKaemmerling

I found a workaround for 2). I enabled the “authorized endpoint” using a load balancer that talks to the masters via private IP. Seems to work. It’s not a big deal if I have to keep Rancher and the clusters in the same project, but it would be nice to separate them into different projects.

vitobotta on Jul 23, 2020

We’ve just set-up a brand new 3 node k3s (1.17) cluster on Hetzner, and hcloud-cloud-controller-manager works perfectly. We have a load balancer set-up as well. We are using networks with it.

We initially had issues with missing prefix hcloud://, but a complete re-deployment (it won’t work with just a restart) with --disable-cloud-controller --kubelet-arg cloud-provider=external fixed it.

MrSaints on Jun 30, 2020

@vitobotta if you find any solution for 1) it would be very nice. We are in a early testing stage with R2 and hetzner cloud at the moment and are hanging here too, since it is not a suitable solution for us to keep clusters in same project. In our case some clusters are not even the same hetzner customers.

ItsReddi on Jul 23, 2020