k3s: k3s agent can't connect to master "failed to get CA certs"

this is a school project, hence the ips are not hidden.

Environmental Info: Master k3s -v k3s version v1.20.2+k3s1 (1d4adb03) go version go1.15.5

Node(s) CPU architecture, OS, and Version: each node is within a AWS t2.micro instances

Describe the bug: It seems that the agent node can’t get the correct TSL certificate in order to communicate with the master node. We are in desperate need for a fix soon, but we can’t seem to solve the issue.

Reproduction:

Master vm: export K3S_DATASTORE_ENDPOINT='mysql://k3s:123qweasdzxc@tcp(34.89.128.178:3306)/k3s'

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig ~/.kube/config --write-kubeconfig-mode 666 external-hostname=3.22.187.191 --tls-san 3.22.187.191 --node-external-ip=3.22.187.191" sh -

sudo kubectl get nodes

NAME              STATUS   ROLES                  AGE   VERSION
ip-172-31-20-43   Ready    control-plane,master   78m   v1.20.2+k3s1

sudo cat /var/lib/rancher/k3s/server/node-token

Agent vm:

curl -sfL https://get.k3s.io | K3S_URL=https://3.22.187.191:6443 K3S_TOKEN=K10c51acd66ae9515cc8260c025fb7d2cf9707762c89f1dde9e02c195dc9b65a0e3::server:85157734a5505d44fbdc96c83f2dc580 sh -

Master vm:

sudo kubectl get nodes
NAME              STATUS   ROLES                  AGE   VERSION
ip-172-31-20-43   Ready    control-plane,master   81m   v1.20.2+k3s1

Agent vm:

sudo systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-01-24 19:22:54 UTC; 47s ago
       Docs: https://k3s.io
    Process: 3722 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 3723 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 3724 (k3s-agent)
      Tasks: 6
     Memory: 171.8M
     CGroup: /system.slice/k3s-agent.service
             └─3724 /usr/local/bin/k3s agent

Jan 24 19:22:54 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:54Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/>
Jan 24 19:22:54 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:54Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/b>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.312499368Z" level=info msg="Starting k3s agent v1.19.5+k3s2 (746>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313312920Z" level=info msg="Module overlay was already loaded"
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313473807Z" level=info msg="Module nf_conntrack was already load>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313612229Z" level=info msg="Module br_netfilter was already load>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313749406Z" level=info msg="Module iptable_nat was already loade>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.315148865Z" level=info msg="Running load balancer 127.0.0.1:4511>
Jan 24 19:23:17 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:23:17.316631450Z" level=error msg="failed to get CA certs: Get \"https>
Jan 24 19:23:39 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:23:39.318819874Z" level=error msg="failed to get CA certs: Get \"https>
lines 1-22/22 (END)

Additional context / logs:

://127.0.0.1:45113/cacerts\": read tcp 127.0.0.1:58444->127.0.0.1:45113: read: connection reset by peer"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 14
  • Comments: 18 (5 by maintainers)

Most upvoted comments

@brandond, I’m having the exact same problem. However, when running the curl command to get the cacerts, it works perfectly fine. I had the exact same problem with RKE2 as well. Any thoughts as to what could be causing this?

Update: When I update /etc/systemd/system/k3s-agent.service.env to use the IP address of the manager node, it works fine. It has issues going through the load balancer for some reason but works fine when I use curl to get the cacerts from that node.