k3s: Failed to start agent (node.crt) 403 forbidden

Agent nodes are now registering with the server using a randomly generated password, where the server will save the password and verify against future attempts.

See /var/lib/rancher/k3s/server/cred/passwd and /var/lib/rancher/k3s/agent/node-password.txt.

We should improve the behavior of this process and/or documentation.

@gonzochic I just simply remove the row for the agent nodes in :/var/lib/rancher/k3s/server/cred/passwd and the agent will try to connect again and success

The node password to remove is in; /var/lib/rancher/k3s/server/cred/passwd/node-passwd

You will see random string followed by node name.

@steve-winter The node password to remove is in; /var/lib/rancher/k3s/server/cred/passwd/node-passwd

Clarification… the file is /var/lib/rancher/k3s/server/cred/node-passwd

Worked like a champ. I had a mix-arch cluster (4 arm64’s and an intel agent). I screwed up Flannel so just decided to reinstall k3s. I ran uninstall script on all 5 nodes succesfully. And the reinstall worked fine on the arm64 master and three workers. But the Intel agent gave me fits until I found this issue.

None of the solutions commented above are working for me (@erikwilson @derekhe). I’ve installed 3 times servers and agents and still the same issue. I have changed the password in the agent to match server’s one and the same. The steps that I followed:

1. Install server on node 1 curl -sfL https://get.k3s.io | sh -

2. Once it finishes, I check token and passwords:

sudo cat  /var/lib/rancher/k3s/server/node-token
K10d0b0a18a5ee13bb5d2d7169b464b7bda4caf7a608aae38570cd0d68c5d7e5a66::node:a98d365cf84b08b10b7423ea55df6563

$ sudo cat /var/lib/rancher/k3s/server/cred/passwd
9e9b5473d1c841d28097a6856bd44152,admin,admin,system:masters
385b9ce5edb6b6cb9f2a026c3997667b,system,system,system:masters
a98d365cf84b08b10b7423ea55df6563,node,node,system:masters

I guess the correct passwd entry is the third one. And it also matches with the part from the token: node:a98d365cf84b08b10b7423ea55df6563. I imagine that is the way the agent knows about the server password.

1. Install agent on node 2 curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.115:6443 K3S_TOKEN=K10d0b0a18a5ee13bb5d2d7169b464b7bda4caf7a608aae38570cd0d68c5d7e5a66::node:a98d365cf84b08b10b7423ea55df6563 sh -

2. journal logs show error authenticating: Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server passwd entry

3. I check the server password in the agent and it is wrong (/var/lib/rancher/k3s/agent/node-password.txt) , so I set it to a98d365cf84b08b10b7423ea55df6563

4. Still the same error as point 4.

Do you have any idea what could be wrong? Thanks 😃

@franciscojsc that is correct. I got both agent and server with the same hostname (raspberry) and it was conflicting. I rename the agent and everything start working again. Is there any place in the documentation specifying that the hostnames have to be different? I could not find it

#359 adds some information to the logs with https://github.com/rancher/k3s/commit/2c9444399b427ffb706818f5bf3892a8880673bf#diff-3bad6c3d4f6253f430990a390e851fdbR87 and https://github.com/rancher/k3s/commit/2c9444399b427ffb706818f5bf3892a8880673bf#diff-c45ec7f2fffc93a75fb235d3efcda580R292, still needs documentation

@erikwilson Thanks for your quick response. Now I understand the current behaviour.

I don’t think it is a bug – it is more a feature which is not described in the documentation. The question which should be answered there: How can we connect a new agent from a machine with a previously installed agent (hence another password).

For now I would simply edit /var/lib/rancher/k3s/agent/node-password.txt to match the password on the server. Not sure if it makes sense to add an additional environment variables (e.g. K3S_AGENT_PASSWORD) since it adds complexity to the public API for an edge-case scenario. It could make sense if you want to make password(s) configurable (e.g. to automatically bootstrap K3S via some other frameworks), but then you would need to add password configuration APIs to the agent and the server.

Encountered the same issue. I believe it happens when you connect an agent, then reinstall the agent and then try to connect again. I fixed the issues by reinstalling the master and reconnecting the agents. Maybe it is enough the rename the hostname of the agent but since installing / uninstalling k3s is so quick I’ve choosen the reinstall.