kubeadm: kubeadm join control-plane node times out (etcd timeout)
What keywords did you search in kubeadm issues before filing this one?
etcd join timeout kubeadm join timeout
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.2”, GitCommit:“f6278300bebbb750328ac16ee6dd3aa7d3549568”, GitTreeState:“clean”, BuildDate:“2019-08-05T09:20:51Z”, GoVersion:“go1.12.5”, Compiler:“gc”, Platform:“linux/amd64”}
Environment:
-
Kubernetes version (use
kubectl version): v1.15.2 -
Cloud provider or hardware configuration: Openstack
-
OS (e.g. from /etc/os-release): Container Linux by CoreOS 2135.5.0 (Rhyolite)
-
Kernel (e.g.
uname -a): Linux os1pi019-kube-master01 4.19.50-coreos-r1 #1 SMP Mon Jul 1 19:07:03 -00 2019 x86_64 Intel® Xeon® CPU E5-2670 v3 @ 2.30GHz GenuineIntel GNU/Linux -
Others:
What happened?
kubeadm join was invoced and failed.
The etcd container did start up 7 seconds after kubeadm timed out / did exit with failure.
See the following logs (this include kubeadm logs and timestamps for pod-manifest starts):
09:30:27 kubeadm service starts
09:30:27 kubeadm[2025]: [preflight] Reading configuration from the cluster...
09:30:27 kubeadm[2025]: [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
09:30:27 kubeadm[2025]: [control-plane] Using manifest folder "/etc/kubernetes/manifests"
09:30:27 kubeadm[2025]: [control-plane] Creating static Pod manifest for "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "audit-policy" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "policy-controller" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "audit-log" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "scheduler-policy" to "kube-scheduler"
09:30:27 kubeadm[2025]: [control-plane] Creating static Pod manifest for "kube-controller-manager"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "audit-policy" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "policy-controller" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "audit-log" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "scheduler-policy" to "kube-scheduler"
09:30:27 kubeadm[2025]: [control-plane] Creating static Pod manifest for "kube-scheduler"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "audit-policy" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "policy-controller" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "audit-log" to "kube-apiserver"
09:30:27 kubeadm[2025]: [controlplane] Adding extra host path mount "scheduler-policy" to "kube-scheduler"
09:30:27 kubeadm[2025]: [check-etcd] Checking that the etcd cluster is healthy
09:30:27 kubeadm[2025]: [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
09:30:27 kubeadm[2025]: [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
09:30:27 kubeadm[2025]: [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
09:30:27 kubeadm[2025]: [kubelet-start] Activating the kubelet service
09:30:27 kubeadm[2025]: [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
09:30:29 kubeadm[2025]: [etcd] Announced new etcd member joining to the existing etcd cluster
09:30:29 kubeadm[2025]: [etcd] Wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
09:30:29 kubeadm[2025]: [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
09:30:38 etcd pause shim create
09:30:30 kube-scheduler pause shim create
09:30:30 kube-controller-manager pause shim create
09:30:34 kube-scheduler shim create
09:30:35 kube-scheduler first logs
09:30:36 kube-apiserver pause shim create
09:31:07 kubeadm[2025]: [kubelet-check] Initial timeout of 40s passed.
09:31:25 kube-controller-manager shim create
09:31:25 kube-controller-manager first logs
09:31:43 kube-apiserver shim create
09:31:44 kubeadm[2025]: error execution phase control-plane-join/etcd: error creating local etcd static pod manifest file: timeout waiting for etcd cluster to be available
09:31:44 systemd[1]: kubeadm.service: Main process exited, code=exited, status=1/FAILURE
09:31:44 kube-apiserver first logs
09:31:51 etcd shim create
09:31:52.081609 etcd first logs
The timeout we hit here is this one which uses hardcoded values (8 times 5 seconds -> 40s)
What you expected to happen?
The etcd member get’s joined to the existing control-plane node and kubeadm succeeds.
How to reproduce it (as minimally and precisely as possible)?
Hard to say.
Try lots of kubeadm joins of control-plane nodes
Anything else we need to know?
In kubeadm init there is a similar looking parameter called TimeoutForControlPlane which defaults to 4 Minutes and is used here to wait for the API server.
This is similar to me because the problem described here and the code at the kubeadm init phase waits for a specific pod, started by the kubelet via a pod manifest.
I see three options:
- increase the hardcoded values
- use the same parameter as already used during init (
TimeoutForControlPlane) which would result in no change to the kubeadm specs - add an additional parameter to the kubeadm spec
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 29 (15 by maintainers)
I belive that I have found the issue to this.
When observing
/etc/kubernetes/manifests/etcd.yamlon the backup master that is trying to join, you will see that it advertises on a different IP range than the primary master.To avoid this, you must specify the advertise address manually when joining:
Where
<ip>is an address in the same subnet as the control plane.