kubeadm: Can't join existing cluster JWS not found?

Trying to join an existing ~20 day old cluster:

sudo kubeadm join --token <redacted>
[preflight] Running pre-flight checks.
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.01.0-ce. Max validated version: 17.03
	[WARNING FileExisting-crictl]: crictl not found in system path
[discovery] Trying to connect to API Server "10.0.0.1:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.0.1:6443"
[discovery] Failed to connect to API Server "10.0.0.1:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "aec65f" is invalid for this cluster, can't connect
[discovery] Trying to connect to API Server "10.0.0.1:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.0.1:6443"
[discovery] Failed to connect to API Server "10.0.0.1:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "aec65f" is invalid for this cluster, can't connect

Any ideas?

Thanks

Cluster was created by kubeadm 1.9.0, current kubeadm is 1.9.2

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 37 (13 by maintainers)

Most upvoted comments

It seem that token expired, Use kubeadm token create on the master node to creating a new valid token

+22

stewart-yu on Jan 23, 2018

Can you show controller-manager log from your master? It looks like cluster-info configmap was not updated for some reason. I hope there will be errors in the controller manager log that would help to investigate why that happened.

bart0sh on Feb 14, 2018

It depends on what version of kubernetes you are running: example 1.18 you can generate new token that supports v=5.

step 1: sudo kubeadm --v=5 token create --print-join-command (this will update cluster-info.yaml with JWS in data section)

step 2: run the above command on node should work. kubeadm join 172.16.26.136:6443 --token 0l27fp.tegcha916hiwn4lv --discovery-token-ca-cert-hash sha256:058073bb05c1d15ec802288c815e2f1d5fa12f912e6e7da9086f4b7c2e2aa850

devtech0101 on Aug 23, 2020

Like I speculated yesterday, this is not a bug related to signing. Rather, it’s a documentation and error exposure failure.

using cloudProvider: aws imposes a number of additional requirements on an environment that aren’t present for non-cloudProvider clusters. Failures are hidden away in secret places: the kubelet logs, the controller manager logs (as seen above), and the api server pod logs.

Here’s how I got everything to work:

I had to set each node’s hostname to a FQDN. On Ubuntu this meant: sudo hostname $(curl 169.254.169.254/latest/meta-data/hostname).
Each node needs an AWS tag that looks like kubernetes.io/cluster/<cluster-name>, set to either owned or `shared.
I gave the master an IAM role that lets it manage clusters.

There’s a good writeup by Nate Baker on these requirements.

@marranz hope this helps!

liztio on Apr 4, 2018

@dixudx creation of a new token did not help and the new configmap also has the same issue (no jws-kubeconfig). I think there should be something unique to the node and am debugging along those lines.

Do you know if there are some packages needed etc for this token to be created on the machine?

EDIT: Apologies @dixudx, this command does work on a test node. I mistook the token create command to mean recreation of the cluster with the new token. I’ll try on the other node of interest and update the thread.

arunmk on Jan 28, 2018

I tried that. The token in the error message changes, but the error persists.

I can see the tokens when I run kubeadm token list on the master, any more things I can do to help debug?

brendandburns on Jan 24, 2018