cluster-api: Can't have 3 master nodes when following 'Quick Start' in book

I followed the instruction in https://cluster-api.sigs.k8s.io/user/quick-start.html with docker infrastructure provider, at the end, I saw 1 master node and 3 worker nodes ready, but I should see 3 master nodes instead.

When I run ‘kubectl describe kubeadmcontrolplane capi-quickstart-control-plane’, I saw a warning in the end:

Events: Type Reason Age From Message

Warning ControlPlaneUnhealthy 4m6s (x1310 over 10h) kubeadm-control-plane-controller Waiting for control plane to pass preflight checks to continue reconciliation: [machine capi-quickstart-control-plane-5skgf does not have APIServerPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have ControllerManagerPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have SchedulerPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have EtcdPodHealthy condition, machine capi-quickstart-control-plane-5skgf does not have EtcdMemberHealthy condition]

Is the warning the reason that only 1 master node is up?

What did you expect to happen: I expect to see 3 master nodes ready (mandatory), and the warning is removed (optional).

Environment:

Cluster-api version: clusterctl version: &version.Info{Major:“0”, Minor:“3”, GitVersion:“v0.3.11”, GitCommit:“e9cf6846b6d93dedadfcf44c00357d15f5ccba64”, GitTreeState:“clean”, BuildDate:“2020-11-19T18:49:17Z”, GoVersion:“go1.13.15”, Compiler:“gc”, Platform:“linux/amd64”}
Minikube/KIND version: kind v0.7.0 go1.13.6 linux/amd64
Kubernetes version: (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“19”, GitVersion:“v1.19.4”, GitCommit:“d360454c9bcd1634cf4cc52d1867af5491dc9c5f”, GitTreeState:“clean”, BuildDate:“2020-11-11T13:17:17Z”, GoVersion:“go1.15.2”, Compiler:“gc”, Platform:“linux/amd64”}
OS (e.g. from /etc/os-release): NAME=“Ubuntu” VERSION=“20.04.1 LTS (Focal Fossa)”

/kind bug /area clusterctl

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (9 by maintainers)

Most upvoted comments

Same issue here: I could see only two docker containers newly created for the workload cluster which means that there was only one master node:

$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                                  NAMES
b962d1e63fe2   kindest/node:v1.18.2           "/usr/local/bin/entr…"   8 minutes ago    Up 8 minutes    44905/tcp, 127.0.0.1:44905->6443/tcp   capi-quickstart-control-plane-fwmrb
2f29fcdb7fd0   kindest/haproxy:2.1.1-alpine   "/docker-entrypoint.…"   9 minutes ago    Up 9 minutes    42317/tcp, 0.0.0.0:42317->6443/tcp     capi-quickstart-lb
...

The weird thing was, after untainting the only master node, the workload cluster was functioning perfectly well. For example, I could access it and deploy workloads there.

$ kubectl get nodes --kubeconfig=./capi-quickstart.kubeconfig
NAME                                  STATUS   ROLES    AGE   VERSION
capi-quickstart-control-plane-h8s2j   Ready    master   19m   v1.18.2

$ kubectl get pods --kubeconfig=./capi-quickstart.kubeconfig
NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-s4kbd   1/1     Running   0          6m11s

But when checking the workload cluster, it’s not yet ready and I could see many errors like what @Insullone reported.

$ kubectl get Machine -A
NAMESPACE   NAME                                    PROVIDERID   PHASE          VERSION
default     capi-quickstart-control-plane-h8s2j                  Provisioning   v1.18.2
default     capi-quickstart-md-0-5477855897-lvblt                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-xhc6x                Pending        v1.18.2
default     capi-quickstart-md-0-5477855897-z7kn8                Pending        v1.18.2

$ kubectl get MachineDeployment -A
NAMESPACE   NAME                   PHASE       REPLICAS   READY   UPDATED   UNAVAILABLE
default     capi-quickstart-md-0   ScalingUp   3                  3         3

$ kubectl get MachineDeployment capi-quickstart-md-0 -o yaml

apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachineDeployment
metadata:
...
spec:
  clusterName: capi-quickstart
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: capi-quickstart
      cluster.x-k8s.io/deployment-name: capi-quickstart-md-0
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: capi-quickstart
        cluster.x-k8s.io/deployment-name: capi-quickstart-md-0
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: KubeadmConfigTemplate
          name: capi-quickstart-md-0
          namespace: default
      clusterName: capi-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: DockerMachineTemplate
        name: capi-quickstart-md-0
        namespace: default
      version: v1.18.2
status:
  observedGeneration: 1
  phase: ScalingUp
  replicas: 3
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart,cluster.x-k8s.io/deployment-name=capi-quickstart-md-0
  unavailableReplicas: 3
  updatedReplicas: 3

brightzheng100 on Dec 20, 2020