kubeadm: Kubeadm join fails with Invalid/expired token error in CoreOS

Versions

kubeadm version (use kubeadm version):

cluster1-2 bin # kubeadm version kubeadm version: &version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:17:50Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:20:10Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:12:17Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”}

• Cloud provider or hardware configuration:

CoreOS VM on OpenShift

• OS (e.g. from /etc/os-release): CoreOS
• Kernel (e.g. uname -a):

Linux cluster1-1.novalocal 4.19.86-coreos #1 SMP Mon Dec 2 20:13:38 -00 2019 x86_64 Intel® Xeon® CPU E5-2680 v3 @ 2.50GHz GenuineIntel GNU/Linux

• Others:

What happened?

I was setting up a new cluster with kubeadm .

In master

1.cat << EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

2 .systemctl restart docker

3. docker info | grep -i cgroup
Cgroup Driver: systemd

4. kubeadm init --pod-network-cidr=10.244.0.0/16

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:
Got the token
kubeadm join 192.168.0.10:6443 --token vw6rtu.yyy\
    --discovery-token-ca-cert-hash sha256:xxx

5. Installed flannel 

In woker node

1.Updated hostname in /etc/hosts
2. Updated docker Cgroup driver
3. cluster1-2 bin # kubeadm join 192.168.0.10:6443 --token vw6rtu.08xn7hqhn2hadvek     --discovery-token-ca-cert-hash sha256:9acfeca4e2d0e420ddf068b7a29f1647ff296211cd204516c3aae755453404ad --v=2
W1218 11:11:43.253700    2155 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
I1218 11:11:43.253779    2155 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName
I1218 11:11:43.253852    2155 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I1218 11:11:43.253977    2155 preflight.go:90] [preflight] Running general checks
I1218 11:11:43.254110    2155 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1218 11:11:43.254129    2155 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf
I1218 11:11:43.254139    2155 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1218 11:11:43.254148    2155 checks.go:102] validating the container runtime
I1218 11:11:43.336428    2155 checks.go:128] validating if the service is enabled and active
I1218 11:11:43.422723    2155 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1218 11:11:43.422921    2155 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1218 11:11:43.423063    2155 checks.go:649] validating whether swap is enabled or not
I1218 11:11:43.423188    2155 checks.go:376] validating the presence of executable ip
I1218 11:11:43.423292    2155 checks.go:376] validating the presence of executable iptables
I1218 11:11:43.423379    2155 checks.go:376] validating the presence of executable mount
I1218 11:11:43.423455    2155 checks.go:376] validating the presence of executable nsenter
I1218 11:11:43.423540    2155 checks.go:376] validating the presence of executable ebtables
I1218 11:11:43.423620    2155 checks.go:376] validating the presence of executable ethtool
I1218 11:11:43.423688    2155 checks.go:376] validating the presence of executable socat
I1218 11:11:43.423767    2155 checks.go:376] validating the presence of executable tc
I1218 11:11:43.423848    2155 checks.go:376] validating the presence of executable touch
I1218 11:11:43.423948    2155 checks.go:520] running all checks
I1218 11:11:43.518141    2155 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I1218 11:11:43.518527    2155 checks.go:618] validating kubelet version
I1218 11:11:43.580704    2155 checks.go:128] validating if the service is enabled and active
I1218 11:11:43.593413    2155 checks.go:201] validating availability of port 10250
I1218 11:11:43.593746    2155 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt
I1218 11:11:43.593811    2155 checks.go:432] validating if the connectivity type is via proxy or direct
I1218 11:11:43.593909    2155 join.go:441] [preflight] Discovering cluster-info
I1218 11:11:43.594077    2155 token.go:188] [discovery] Trying to connect to API Server "192.168.0.10:6443"
I1218 11:11:43.594961    2155 token.go:73] [discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.10:6443"
I1218 11:11:43.604812    2155 token.go:191] [discovery] Failed to connect to API Server "192.168.0.10:6443": token id "vw6rtu" is invalid for this cluster or it has expired. Use "kubeadm token create" on the control-plane node to create a new valid token
I1218 11:11:48.605151    2155 token.go:188] [discovery] Trying to connect to API Server "192.168.0.10:6443"
I1218 11:11:48.605631    2155 token.go:73] [discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.10:6443"
I1218 11:11:48.607687    2155 token.go:191] [discovery] Failed to connect to API Server "192.168.0.10:6443": token id "vw6rtu" is invalid for this cluster or it has expired. Use "kubeadm token create" on the control-plane node to create a new valid token
I1218 11:11:53.607925    2155 token.go:188] [discovery] Trying to connect to API Server "192.168.0.10:6443"
I1218 11:11:53.608882    2155 token.go:73] [discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.10:6443"
I1218 11:11:53.611365    2155 token.go:191] [discovery] Failed to connect to API Server "192.168.0.10:6443": token id "vw6rtu" is invalid for this cluster or it has expired. Use "kubeadm token create" on the control-plane node to create a new valid token

This is not a problem with token expirty; and firewalls are open for 6443

Still in master node again

$ kubeadm token create --ttl 0 --print-join-command
W1218 11:19:12.284747    1975 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1218 11:19:12.284786    1975 validation.go:28] Cannot validate kubelet config - no validator is available
kubeadm join 192.168.0.10:6443 --token uwa2zs.g4zibdv54mbcjs32     --discovery-token-ca-cert-hash sha256:9acfeca4e2d0e420ddf068b7a29f1647ff296211cd204516c3aae755453404ad

In node I want to join

kubeadm join 192.168.0.10:6443 --token uwa2zs.g4zibdv54mbcjs32     --discovery-token-ca-cert-hash sha256:9acfeca4e2d0e420ddf068b7a29f1647ff296211cd204516c3aae755453404ad --v=4

Same error

2 token.go:191] [discovery] Failed to connect to API Server "192.168.0.10:6443": token id "uwa2zs" is invalid for this cluster or it has expired. Use "kubeadm token create" on the control-plane node to create a new valid token

What you expected to happen?

kubeadm join should work

How to reproduce it (as minimally and precisely as possible)?

Steps given

Anything else we need to know?