kubeadm: Marking the master by adding the taints 'error marking master: timed out waiting for the condition'

What keywords did you search in kubeadm issues before filing this one?

error marking master: timed out waiting for the condition

https://github.com/kubernetes/kubeadm/issues/1092

https://github.com/kubernetes/kubeadm/issues/937

https://github.com/kubernetes/kubeadm/issues/1087

https://github.com/kubernetes/kubeadm/issues/715

https://github.com/kubernetes/kubernetes/issues/45727

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:51:33Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

kubectl version --kubeconfig=/etc/kubernetes/admin.conf
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

ubuntu xenial on baremetal

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

Linux testymaster1 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

kubeadm init --config /etc/kubernetes/kubeadmcfg.yaml
[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [testymaster1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.23.238 10.0.23.238 127.0.0.1 10.0.23.241 10.0.23.242 10.0.23.243 10.0.23.238 10.0.23.239 10.0.23.240 10.0.23.244 10.0.23.245 10.0.23.246]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"                                                                                                                      
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"                                                                                                    
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"                                                                                                                      
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 22.506258 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster                                                                                                          
[markmaster] Marking the node testymaster1 as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node testymaster1 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
error marking master: timed out waiting for the condition

What you expected to happen?

Master to initialize without issue.

How to reproduce it (as minimally and precisely as possible)?

https://gist.github.com/joshuacox/4505fbeceb2e394900a24c3cae14131c

run the above like so:

bash etcd-test6.sh 10.0.0.6 10.0.0.7 10.0.0.8'

at this point you should have a healthy etcd cluster running on three hosts

then on a separate host (10.0.0.9) run the steps detailed here:

https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd

with this config:

apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
apiServerCertSANs:
- "127.0.0.1"
- '10.0.0.6'
- '10.0.0.7'
- '10.0.0.8'
- '10.0.0.9'
controlPlaneEndpoint: "10.0.0.9"
etcd:
  external:
      endpoints:
      - https://10.0.0.6:2379
      - https://10.0.0.7:2379
      - https://10.0.0.8:2379
      caFile: /etc/kubernetes/pki/etcd/ca.crt
      certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
      keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
networking:
  podSubnet: 10.244.0.0/16

Anything else we need to know?

CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
cbc9036b0675        51a9c329b7c5           "kube-apiserver --..."   23 minutes ago      Up 23 minutes                           k8s_kube-apiserver_kube-apiserver-testymaster1_kube-system_c55b3dd53dd51e69d2acd3a6aa486e32_0
aeebe73a2c98        d6d57c76136c           "kube-scheduler --..."   23 minutes ago      Up 23 minutes                           k8s_kube-scheduler_kube-scheduler-testymaster1_kube-system_ee7b1077c61516320f4273309e9b4690_0
58fc131c3b50        15548c720a70           "kube-controller-m..."   23 minutes ago      Up 23 minutes                           k8s_kube-controller-manager_kube-controller-manager-testymaster1_kube-system_690790d9ba49d9118c24c004854af4db_0
4f628d299b8e        k8s.gcr.io/pause:3.1   "/pause"                 23 minutes ago      Up 23 minutes                           k8s_POD_kube-scheduler-testymaster1_kube-system_ee7b1077c61516320f4273309e9b4690_0
2fe08cdd58c9        k8s.gcr.io/pause:3.1   "/pause"                 23 minutes ago      Up 23 minutes                           k8s_POD_kube-controller-manager-testymaster1_kube-system_690790d9ba49d9118c24c004854af4db_0
85638811980c        k8s.gcr.io/pause:3.1   "/pause"                 23 minutes ago      Up 23 minutes                           k8s_POD_kube-apiserver-testymaster1_kube-system_c55b3dd53dd51e69d2acd3a6aa486e32_0

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf, 20-etcd-service-manager.conf
   Active: active (running) since Sun 2018-11-11 20:09:14 UTC; 19min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 4731 (kubelet)
    Tasks: 60
   Memory: 40.2M
      CPU: 59.614s
   CGroup: /system.slice/kubelet.service
           └─4731 /usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true

Nov 11 20:27:54 testymaster1 kubelet[4731]: I1111 20:27:54.434903    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:27:58 testymaster1 kubelet[4731]: I1111 20:27:58.434922    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:00 testymaster1 kubelet[4731]: I1111 20:28:00.737709    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:10 testymaster1 kubelet[4731]: I1111 20:28:10.788482    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:18 testymaster1 kubelet[4731]: I1111 20:28:18.434933    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:20 testymaster1 kubelet[4731]: I1111 20:28:20.828593    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:30 testymaster1 kubelet[4731]: I1111 20:28:30.877710    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:40 testymaster1 kubelet[4731]: I1111 20:28:40.924675    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:28:50 testymaster1 kubelet[4731]: I1111 20:28:50.974638    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 11 20:29:01 testymaster1 kubelet[4731]: I1111 20:29:01.024980    4731 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach

journalctl -xeu kubelet

https://gist.github.com/joshuacox/3c0b4aa2b66d1172067a32e6e064f948

docker logs cbc9036b0675 the kube api container logs:

https://gist.github.com/joshuacox/ab29412c1653e2b1fd2fa06cdd0ae2e2