cluster-api: Creating control plane machine following the quick start guide fails with "failed to retrieve kubeconfig secret"

What steps did you take and what happened: After following the instructions in the quick start guide and using the Machine object definition defined under the usage section for vSphere the VM is created successfully but the <cluster>-kubeconfig secret is not created. On further troubleshooting, the capi-controller-manager logs clearly show that it is unable to retrieve this secret

root@cli-vm:~# k logs capi-controller-manager-6c64c695bb-gwkj5 -n capi-system

E0220 15:36:11.664898       1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" 
"error"="failed to retrieve kubeconfig secret for Cluster \"capi-quickstart\" in namespace \"default\": Secret
 \"capi-quickstart-kubeconfig\" not found"  "controller"="machine" "request"=
{"Namespace":"default","Name":"capi-quickstart-controlplane-0"}

From the capv-controller-manager logs the error message says that the KubeadmConfig.bootstrap.cluster.x-k8s.io \"capi-quickstart-controlplane-0\" not found" which is very strange as the object is created successfully.

k logs -n capv-system capv-controller-manager-88f646758-pr6wj

E0220 15:27:23.411515       1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" 
"error"="failed to reconcile API endpoints for VSphereCluster default/capi-quickstart: failed to get 
KubeadmConfig capi-quickstart-controlplane-0/ for Machine default//capi-quickstart-controlplane-0: 
KubeadmConfig.bootstrap.cluster.x-k8s.io \"capi-quickstart-controlplane-0\" not found"  
"controller"="vspherecluster" "request"={"Namespace":"default","Name":"capi-quickstart"}

What did you expect to happen: Once the cluster and machine objects are created, the <cluster-name>-kubeconfig secret should also be created successfully.

Anything else you would like to add: After looking at the error message, I could see that the quick start guide’s KubeadmConfig object is missing the namespace property as expected in the kubeadmConfigKey’s namespace definition machine.Spec.Bootstrap.ConfigRef.Namespace

Adding the namespace resolved the issue. I’ll submit a PR to fix the docs to include the namespace but given the namespace is not a required field I expected that the kubeconfig secret would have generated successfully.

Environment:

Cluster-api version: v1alpha2
KIND version: 0.7.0
Kubernetes version: (use kubectl version): 1.16.3
OS (e.g. from /etc/os-release): Ubuntu

/kind bug /kind documentation

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (19 by maintainers)

Most upvoted comments

Closing given that the CAPV issue has been closed, feel free to reopen if necessary.

/close

vincepri on Jun 24, 2020

I’d rather not, as that makes it so the QuickStart only works in the default namespace. Let’s fix CAPV to resolve this issue.

ncdc on Feb 24, 2020

OK, here is what I believe is happening:

From the capv-controller-manager logs the error message says that the KubeadmConfig.bootstrap.cluster.x-k8s.io "capi-quickstart-controlplane-0" not found" which is very strange as the object is created successfully.

This is a temporary “error” that is logged because the KubeadmConfig is not in the controller’s cache for a period of time. It eventually goes away. Safe to ignore.

controller-runtime/controller "msg"="Reconciler error" "error"="failed to retrieve kubeconfig secret for Cluster \"capi-quickstart\" in namespace \"default\": Secret \"capi-quickstart-kubeconfig\" not found" "controller"="machine" "request"= {"Namespace":"default","Name":"capi-quickstart-controlplane-0"}

Note the case of kubeconfig in failed to retrieve kubeconfig secret - it’s all lowercase. This comes from remote.NewClusterClient(). This function is used in CAPI in 4 places:

machine_controller.go#deleteNode
machine_controller.go#drainNode
machine_controller_noderef.go#reconcileNodeRef
machineset_status.go#getMachineNode

Of these 4, I’m reasonably certain we’re dealing with the reconcileNodeRef case. We can’t reconcile the node ref until we have a kubeconfig secret for the workload cluster. And we can’t create that secret until cluster.status.apiEndpoints is set. For CAPV, the API endpoints are set either from an annotation on the VSphereCluster, or after the first control plane machine is running. You may be running into a situation where you need to wait a bit longer for the kubeconfig secret to be created (once the API endpoints are set).

ncdc on Feb 24, 2020