cluster-api-provider-hetzner: controlplane bootstrap fails/stuck after first controlplane node

/kind bug

What steps did you take and what happened: i tried to deploy a k8s cluster by using the Hetzner capi provider but the control plane is not able to get healthy. the deployment stuck after the first control plane.

my steps:

clusterctl generate provider --infrastructure hetzner:v1.0.0-alpha.20 > hetzner-capi.yml
kubectl apply -f hetzner-capi.yml

export HCLOUD_TOKEN=xxxxxxxxxxxx
export HCLOUD_SSH_KEY="mykey"
export CLUSTER_NAME="mycluster"
export HCLOUD_REGION="fsn1"
export CONTROL_PLANE_MACHINE_COUNT=3
export WORKER_MACHINE_COUNT=3
export KUBERNETES_VERSION=1.24.1
export HCLOUD_CONTROL_PLANE_MACHINE_TYPE=cpx31
export HCLOUD_WORKER_MACHINE_TYPE=cpx31

kubectl create secret generic hetzner --from-literal=hcloud=${HCLOUD_TOKEN}
kubectl patch secret hetzner-p '{"metadata":{"labels":{"clusterctl.cluster.x-k8s.io/move":""}}}'
clusterctl generate cluster --infrastructure hetzner:v1.0.0-alpha.20 ${CLUSTER_NAME} > ${CLUSTER_NAME}.yaml
kubectl apply -f ${CLUSTER_NAME}.yaml

cluster is created but the controlplane is not able to get healthy:

$ clusterctl describe cluster ${CLUSTER_NAME}
NAME                                                                       READY  SEVERITY  REASON                       SINCE  MESSAGE                                                                  
Cluster/mycluster                                                          False  Warning   ScalingUp                    71m    Scaling up control plane to 3 replicas (actual 1)                        
├─ClusterInfrastructure - HetznerCluster/mycluster                                                                                                                                                       
├─ControlPlane - KubeadmControlPlane/mycluster-control-plane               False  Warning   ScalingUp                    71m    Scaling up control plane to 3 replicas (actual 1)                        
│ └─Machine/mycluster-control-plane-x5jb5                                  False  Warning   NodeStartupTimeout           49m    Node failed to report startup in &Duration{Duration:20m0s,}              
│   └─MachineInfrastructure - HCloudMachine/mycluster-control-plane-prxwq                                                                                                                                
└─Workers                                                                                                                                                                                                
  └─MachineDeployment/mycluster-md-0                                       False  Warning   WaitingForAvailableMachines  72m    Minimum availability requires 3 replicas, current 0 available            
    └─3 Machines...                                                        True                                          7m53s  See mycluster-md-0-59f5696b48-khjkp, mycluster-md-0-59f5696b48-v57kg, ...

$ kubectl get KubeadmControlPlane
NAME                          CLUSTER         INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
mycluster-control-plane       mycluster       true                                 1                  1         1             73m   v1.24.1

$ kubectl describe KubeadmControlPlane mycluster-control-plane
.....
Events:
  Type     Reason                 Age                    From                              Message
  ----     ------                 ----                   ----                              -------
  Warning  ControlPlaneUnhealthy  3m28s (x285 over 73m)  kubeadm-control-plane-controller  Waiting for control plane to pass preflight checks to continue reconciliation: [machine mycluster-control-plane-x5jb5 does not have APIServerPodHealthy condition, machine mycluster-control-plane-x5jb5 does not have ControllerManagerPodHealthy condition, machine mycluster-control-plane-x5jb5 does not have SchedulerPodHealthy condition, machine mycluster-control-plane-x5jb5 does not have EtcdPodHealthy condition, machine mycluster-control-plane-x5jb5 does not have EtcdMemberHealthy condition]

$ kubectl get MachineHealthCheck
NAME                                       CLUSTER         EXPECTEDMACHINES   MAXUNHEALTHY   CURRENTHEALTHY   AGE
mycluster-control-plane-unhealthy-5m       mycluster       1                  100%                            74m
mycluster-md-0-unhealthy-5m                mycluster       3                  100%                            74m

$ kubectl describe MachineHealthCheck mycluster-control-plane-unhealthy-5m
Events:
  Type     Reason          Age                 From                           Message
  ----     ------          ----                ----                           -------
  Warning  ReconcileError  75m (x13 over 75m)  machinehealthcheck-controller  error creating client and cache for remote cluster: error fetching REST client config for remote cluster "default/mycluster": failed to retrieve kubeconfig secret for Cluster default/mycluster: secrets "mycluster-kubeconfig" not found
  Warning  ReconcileError  74m                 machinehealthcheck-controller  error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "default/mycluster": Get "https://142.132.240.114:443/api?timeout=10s": dial tcp 142.132.240.114:443: i/o timeout
  Warning  ReconcileError  73m (x4 over 74m)   machinehealthcheck-controller  error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "default/mycluster": context deadline exceeded

$ kubectl get secrets
NAME                                TYPE                                  DATA   AGE
default-token-vcccj                 kubernetes.io/service-account-token   3      5h35m
hetzner                             Opaque                                1      78m
mycluster-ca                        cluster.x-k8s.io/secret               2      76m
mycluster-control-plane-xkvnf       cluster.x-k8s.io/secret               2      76m
mycluster-etcd                      cluster.x-k8s.io/secret               2      76m
mycluster-kubeconfig                cluster.x-k8s.io/secret               1      76m
mycluster-md-0-48wdl                cluster.x-k8s.io/secret               2      12m
mycluster-md-0-rlccb                cluster.x-k8s.io/secret               2      13m
mycluster-md-0-x2q8s                cluster.x-k8s.io/secret               2      12m
mycluster-proxy                     cluster.x-k8s.io/secret               2      76m
mycluster-sa                        cluster.x-k8s.io/secret               2      76m

# Get the node status of the deployed cluster
$ kubectl get no --kubeconfig mycluster
NAME                            STATUS   ROLES              AGE    VERSION
mycluster-control-plane-x5jb5   NotReady    control-plane   74m    v1.24.1
mycluster-md-0-khjkp            NotReady    <none>          72m    v1.24.1
mycluster-md-0-v57kg            NotReady    <none>          72m    v1.24.1
mycluster-md-0-x5d32            NotReady    <none>          72m    v1.24.1

# Try to fetch API - possible
$ curl https://142.132.240.114:443/api?timeout=10s -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/api\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

If tested hetzner:v1.0.0-alpha.19 and hetzner:v1.0.0-alpha.20 but i get the same result.

Environment:

  • cluster-api-provider-hetzner version: v1.0.0-alpha.19 and v1.0.0-alpha.20
  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"42a9a90338d705a1650fb68b7891f84b62adb0b0", GitTreeState:"clean", BuildDate:"2022-06-15T04:25:21Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
  • OS:
    • Server image: ubuntu 20.04
    • client: OSX

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 22 (12 by maintainers)

Most upvoted comments

then add a sentence that both CNI and CCM are required after the first control plane is ready, otherwise the cluster won’t be usable

With the hcloud-cloud-controller-manager you need to make sure that the correct secret is configured to access the Hetzner Cloud API. In the default configuration the secret created by cluster-api-provider-hetzner in the workload cluster is called hetzner. hcloud-ccm expects a secret called hcloud by default, but can be configured to use whatever secret you want.

I have my cluster running with CAPH & hcloud-ccm and can confirm that it works well for Hetzner-Cloud only scenarios. The logs indicate invalid credentials & issues while talking to the Hetzner Robot API. During the timeframe indicated by the Logs, there was a scheduled maintenance of the Robot API, which would explain the issues: https://status.hetzner.com/incident/e162e98c-61b8-4a5d-8ea8-1ea3a84a6ee1

could you ssh to the node and do crictl ps? And did you installed the ccm and a cni? I see that the node status is NotReady A kubectl get pods -A could be very helpful.