hcloud-cloud-controller-manager: Hcloud manager errors: Couldn't reconcile node routes error listing routes context deadline exceeded
We are using hcloud manager in cluster deployed on Hetzner VMs. Hcloud manager is deployed with network support. After few days it started hit the hetzner cloud api limits and log the following errors :
E0830 16:20:57.819595 1 route_controller.go:118] Couldn't reconcile node routes: error listing routes: hcloud/ListRoutes: hcloud/hcloudRouteToRoute: hcops/AllServersCache.ByPrivateIP: 192.168.1.7 hcops/AllServersCache.getCache: Get "https://api.hetzner.cloud/v1/servers?page=1&per_page=50": context deadline exceeded
Hcloud manager is deployed with kubespray addons and without some specific configurations. It doesnt look to effect the cluster somehow for now , but from the logs it looks like an issue and effects our terraform commands even that they are using different api keys for hetzner cloud api.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 21 (8 by maintainers)
@ym you will get an answer 😃
We want to debug this even further. With one of the last releases, we got a contribution that added metrics to all API calls (https://github.com/hetznercloud/hcloud-cloud-controller-manager/pull/303). You should be able to see how often specific endpoints were called by looking at the metrics of the CCM. Can you send us maybe a screenshot from your grafana dashboard - or if possible - send us access to this dashboard via mail to lukas.kaemmerling(at)hetzner-cloud.de ?
@mmpetarpeshev
Ok, good news, so its the API limit. Dont worry, this is an internal mechanism to prevent spam. I will talk to some collegues how we gonna proceed with those cases since youre not the only one having trouble with it.
Waiting for your final update, i will also keep u up to date 😃