k3s: k3s agent can't connect to master "failed to get CA certs"
this is a school project, hence the ips are not hidden.
Environmental Info:
Master
k3s -v k3s version v1.20.2+k3s1 (1d4adb03) go version go1.15.5
Node(s) CPU architecture, OS, and Version: each node is within a AWS t2.micro instances
Describe the bug: It seems that the agent node can’t get the correct TSL certificate in order to communicate with the master node. We are in desperate need for a fix soon, but we can’t seem to solve the issue.
Reproduction:
Master vm:
export K3S_DATASTORE_ENDPOINT='mysql://k3s:123qweasdzxc@tcp(34.89.128.178:3306)/k3s'
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig ~/.kube/config --write-kubeconfig-mode 666 external-hostname=3.22.187.191 --tls-san 3.22.187.191 --node-external-ip=3.22.187.191" sh -
sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-20-43 Ready control-plane,master 78m v1.20.2+k3s1
sudo cat /var/lib/rancher/k3s/server/node-token
Agent vm:
curl -sfL https://get.k3s.io | K3S_URL=https://3.22.187.191:6443 K3S_TOKEN=K10c51acd66ae9515cc8260c025fb7d2cf9707762c89f1dde9e02c195dc9b65a0e3::server:85157734a5505d44fbdc96c83f2dc580 sh -
Master vm:
sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-20-43 Ready control-plane,master 81m v1.20.2+k3s1
Agent vm:
sudo systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-01-24 19:22:54 UTC; 47s ago
Docs: https://k3s.io
Process: 3722 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 3723 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 3724 (k3s-agent)
Tasks: 6
Memory: 171.8M
CGroup: /system.slice/k3s-agent.service
└─3724 /usr/local/bin/k3s agent
Jan 24 19:22:54 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:54Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/>
Jan 24 19:22:54 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:54Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/b>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.312499368Z" level=info msg="Starting k3s agent v1.19.5+k3s2 (746>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313312920Z" level=info msg="Module overlay was already loaded"
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313473807Z" level=info msg="Module nf_conntrack was already load>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313612229Z" level=info msg="Module br_netfilter was already load>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.313749406Z" level=info msg="Module iptable_nat was already loade>
Jan 24 19:22:57 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:22:57.315148865Z" level=info msg="Running load balancer 127.0.0.1:4511>
Jan 24 19:23:17 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:23:17.316631450Z" level=error msg="failed to get CA certs: Get \"https>
Jan 24 19:23:39 ip-172-31-4-28 k3s[3724]: time="2021-01-24T19:23:39.318819874Z" level=error msg="failed to get CA certs: Get \"https>
lines 1-22/22 (END)
Additional context / logs:
://127.0.0.1:45113/cacerts\": read tcp 127.0.0.1:58444->127.0.0.1:45113: read: connection reset by peer"
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 14
- Comments: 18 (5 by maintainers)
@brandond, I’m having the exact same problem. However, when running the curl command to get the cacerts, it works perfectly fine. I had the exact same problem with RKE2 as well. Any thoughts as to what could be causing this?
Update: When I update /etc/systemd/system/k3s-agent.service.env to use the IP address of the manager node, it works fine. It has issues going through the load balancer for some reason but works fine when I use curl to get the cacerts from that node.