k3s: K3s Install on Raspberry Pi 4b failed (TLS Handshake Timeout pi3, pi4, etc)

Thanks for helping us to improve k3s! We welcome all bug reports. Please fill out each area of the template so we can better help you. You can delete this message portion of the bug report.

Version: Provide the output from k3s -v and provide the flags used to install or run k3s server.

root@raspberrypi:/home/pi# k3s -v
k3s version v0.10.0 (f9888ca3)

OS version: Linux raspberrypi 4.19.75-v7l+ rancher/k3s#1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l bootloader version:

root@raspberrypi:~# vcgencmd bootloader_version
Sep 10 2019 10:41:50
version f626c772b15ba1b7e0532a8d50a761b3ccbdf3bb (release)
timestamp 1568112110

Describe the bug A clear and concise description of what the bug is. After run install command “curl -sfL https://get.k3s.io | sh -”, installation can’t be completed, and TLS handshake timeout error prompted

To Reproduce Steps to reproduce the behavior: Run command ‘curl -sfL https://get.k3s.io | sh -’ on Raspberry Pi 4b 4G memory

Expected behavior A clear and concise description of what you expected to happen.

Actual behavior A clear and concise description of what actually happened. TLS handshake timeout

Additional context Add any other context about the problem here. I put some error logs below, hope them can help:

root@raspberrypi:/home/pi# journalctl -u k3s.service
-- Logs begin at Thu 2019-09-26 01:24:23 BST, end at Sun 2019-10-27 01:22:17 GMT. --
Oct 27 01:19:58 raspberrypi systemd[1]: Starting Lightweight Kubernetes...
Oct 27 01:19:58 raspberrypi k3s[3688]: time="2019-10-27T01:19:58Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/3f43b16ca97dbb7ba58868cdb2137a72ad7215762a2852ed944237bf45d44f07"
Oct 27 01:20:13 raspberrypi k3s[3688]: time="2019-10-27T01:20:13.437098936Z" level=info msg="Starting k3s v0.10.0 (f9888ca3)"
Oct 27 01:20:13 raspberrypi k3s[3688]: time="2019-10-27T01:20:13.945042885Z" level=info msg="Kine listening on unix://kine.sock"
Oct 27 01:20:13 raspberrypi k3s[3688]: time="2019-10-27T01:20:13.947965657Z" level=info msg="Fetching bootstrap data from etcd"
Oct 27 01:20:15 raspberrypi k3s[3688]: time="2019-10-27T01:20:15.186636567Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --basic-auth-file=/var/lib
Oct 27 01:20:15 raspberrypi k3s[3688]: Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
Oct 27 01:20:15 raspberrypi k3s[3688]: I1027 01:20:15.189751    3688 server.go:650] external host was not specified, using 192.168.199.80
Oct 27 01:20:15 raspberrypi k3s[3688]: I1027 01:20:15.191063    3688 server.go:162] Version: v1.16.2-k3s.1
Oct 27 01:20:19 raspberrypi k3s[3688]: I1027 01:20:19.782703    3688 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultT
Oct 27 01:20:19 raspberrypi k3s[3688]: I1027 01:20:19.782801    3688 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeCl
Oct 27 01:20:19 raspberrypi k3s[3688]: I1027 01:20:19.785373    3688 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultT
Oct 27 01:20:19 raspberrypi k3s[3688]: I1027 01:20:19.785425    3688 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeCl
Oct 27 01:20:19 raspberrypi k3s[3688]: I1027 01:20:19.856982    3688 master.go:259] Using reconciler: lease
Oct 27 01:20:19 raspberrypi k3s[3688]: I1027 01:20:19.966350    3688 rest.go:115] the default service ipfamily for this cluster is: IPv4
Oct 27 01:20:20 raspberrypi k3s[3688]: W1027 01:20:20.788011    3688 genericapiserver.go:404] Skipping API batch/v2alpha1 because it has no resources.
Oct 27 01:20:20 raspberrypi k3s[3688]: W1027 01:20:20.853703    3688 genericapiserver.go:404] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Oct 27 01:20:20 raspberrypi k3s[3688]: W1027 01:20:20.919549    3688 genericapiserver.go:404] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Oct 27 01:20:20 raspberrypi k3s[3688]: W1027 01:20:20.931880    3688 genericapiserver.go:404] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Oct 27 01:20:20 raspberrypi k3s[3688]: W1027 01:20:20.973747    3688 genericapiserver.go:404] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Oct 27 01:20:21 raspberrypi k3s[3688]: W1027 01:20:21.043638    3688 genericapiserver.go:404] Skipping API apps/v1beta2 because it has no resources.
Oct 27 01:20:21 raspberrypi k3s[3688]: W1027 01:20:21.043695    3688 genericapiserver.go:404] Skipping API apps/v1beta1 because it has no resources.
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.078307    3688 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultT
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.078434    3688 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeCl
Oct 27 01:20:21 raspberrypi k3s[3688]: time="2019-10-27T01:20:21.096613858Z" level=info msg="Running kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --secure-port=0"
Oct 27 01:20:21 raspberrypi k3s[3688]: time="2019-10-27T01:20:21.098945424Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/s
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.119387    3688 controllermanager.go:161] Version: v1.16.2-k3s.1
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.121660    3688 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.127479    3688 server.go:143] Version: v1.16.2-k3s.1
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.127709    3688 defaults.go:91] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
Oct 27 01:20:21 raspberrypi k3s[3688]: W1027 01:20:21.139439    3688 authorization.go:47] Authorization is disabled
Oct 27 01:20:21 raspberrypi k3s[3688]: W1027 01:20:21.139494    3688 authentication.go:79] Authentication is disabled
Oct 27 01:20:21 raspberrypi k3s[3688]: I1027 01:20:21.139527    3688 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
Oct 27 01:20:31 raspberrypi k3s[3688]: time="2019-10-27T01:20:31.111017958Z" level=fatal msg="starting tls server: Get https://127.0.0.1:6444/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions: net/http: TLS handshake timeout"
Oct 27 01:20:31 raspberrypi systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 01:20:31 raspberrypi systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 27 01:20:31 raspberrypi systemd[1]: Failed to start Lightweight Kubernetes.
Oct 27 01:20:36 raspberrypi systemd[1]: k3s.service: Service RestartSec=5s expired, scheduling restart.
Oct 27 01:20:36 raspberrypi systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1.
Oct 27 01:20:36 raspberrypi systemd[1]: Stopped Lightweight Kubernetes.
Oct 27 01:20:36 raspberrypi systemd[1]: Starting Lightweight Kubernetes...

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 58 (18 by maintainers)

Most upvoted comments

Workaround is to downgrade curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 sh -, thanks to https://twitter.com/ibuildthecloud/status/1188640874642563072

+15

kaihendry on Oct 28, 2019

There’s a race condition happening starting the apiserver and waiting for crds to be created. In pkg/server/context.go:41 the call to create crds is failing because of a timeout waiting for crds in pkg/server/context.go:69. The CRDs is taking time because of the apiserver is not yet available. If adding a simple sleep (not a suggested solution) after pkg/daemons/control/server.go:89 seems to resolve the issue.

larmog on Nov 4, 2019

#1007 is available in v0.11.0-alpha1 to work around the TLS handshake timeout issue.

erikwilson on Nov 8, 2019

v1.17.4+k3s1 does not work for me in similar circumstances, so this isn’t fixed in modern releases.

rcarmo on May 1, 2020

I have the same problem with raspberrypi model 3B+ (version k3s 0.10.0) but with the 0.9.1 it’s working.

mordredz on Oct 28, 2019

$ k3s --version k3s version v0.11.0-alpha2 (405f85a)

failed on RPI3.

INFO[2019-11-10T12:46:55.473870979Z] Done waiting for CRD helmcharts.helm.cattle.io to become available 
FATA[2019-11-10T12:46:55.476566942Z] starting tls server: timed out waiting for the condition

@xiaods I have install rng-tool and set the swapaccount=1 as @zimme comment May it make the difference because is working

# k3s --version
k3s version v0.11.0-alpha2 (405f85aa)
# kubectl get pod -A
NAMESPACE      NAME                                      READY   STATUS      RESTARTS   AGE
kube-system    metrics-server-6d684c7b5-sjh44            1/1     Running     0          132m
kube-system    local-path-provisioner-58fb86bdfd-f4cjr   1/1     Running     0          132m
kube-system    coredns-d798c9dd-8wj8x                    1/1     Running     0          132m
kube-system    helm-install-traefik-pwp9g                0/1     Completed   0          132m
kube-system    svclb-traefik-h7tcv                       3/3     Running     0          131m
kube-system    traefik-65bccdc4bd-vt9hd                  1/1     Running     0          131m
cert-manager   cert-manager-687f47b874-x4jk5             1/1     Running     0          124m
cert-manager   cert-manager-cainjector-f44b4b959-h27xh   1/1     Running     0          124m
cert-manager   cert-manager-webhook-7f8bdb755f-qqcw4     1/1     Running     1          124m
tick           influxdb-deployment-c7cb599b4-txgh5       1/1     Running     0          90m
tick           chronograf-deployment-7c48d8b5dc-c72jf    1/1     Running     0          84m
tick           telegraf-deployment-889755bb-sgkfs        1/1     Running     0          82m
tick           kapacitor-deployement-6cff699c4d-bv8jh    1/1     Running     6          86m

pierremahot on Nov 10, 2019

Unfortunately I feel like arm is completely broken, here is a small change which seems to consistently sigsegv https://drone-pr.rancher.io/rancher/k3s/1820, and I have seen similar small changes in code cause completely unrelated panics in go 1.13 which is why we downgraded to go 1.12. I think there are a few possible problems:

golang is broke (probably for all of arm) network stack is broke (probably for rpi3) kernel is broke (probably only specific versions, maybe unrelated)

but on the same OS, old k3s works well .

thomasdba on Nov 7, 2019

Same here. Couldn’t get any v0.10.x working on rpi3b+ with raspbian up-to-date (even with cgroup_memory=1 cgroup_enable=memory)

mcanevet on Nov 2, 2019

Also tried to install old version v0.9.1, first time failed with cgroup error:

Oct 31 14:09:12 ubuntu k3s[2377]: time="2019-10-31T14:09:12.021942176Z" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Oct 31 14:09:12 ubuntu k3s[2377]: time="2019-10-31T14:09:12.022021433Z" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"

After add the two option of cgroup into /boot/firmware/config.txt file, and tried again, it succeed.

root@ubuntu:~# kubectl get node
NAME     STATUS   ROLES    AGE     VERSION
ubuntu   Ready    master   3m55s   v1.15.4-k3s.1
root@ubuntu:~# kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-5b8648d6f6-7fgm5   1/1     Running     0          3m52s
kube-system   coredns-66f496764-cjg7q                   1/1     Running     0          3m52s
kube-system   helm-install-traefik-szt4n                0/1     Completed   0          3m52s
kube-system   svclb-traefik-9b7cv                       3/3     Running     0          51s
kube-system   traefik-d869575c8-4gf95                   1/1     Running     0          51s

After that, I tried to upgrade K3s to latest version, it succeed this time:

root@ubuntu:~# k3s -version
k3s version v0.9.1 (755bd1c6)
root@ubuntu:~# curl -sfL https://get.k3s.io | sh -
[INFO]  Finding latest release
[INFO]  Using v0.10.1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v0.10.1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v0.10.1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
root@ubuntu:~# kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
ubuntu   Ready    master   7m    v1.15.4-k3s.1
root@ubuntu:~# kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   helm-install-traefik-szt4n                0/1     Completed   0          6m53s
kube-system   local-path-provisioner-5b8648d6f6-7fgm5   0/1     Error       0          6m53s
kube-system   coredns-66f496764-cjg7q                   1/1     Running     0          6m53s
kube-system   svclb-traefik-9b7cv                       3/3     Running     0          3m52s
kube-system   traefik-d869575c8-4gf95                   0/1     Running     0          3m52s
root@ubuntu:~# kubectl get node
NAME     STATUS   ROLES    AGE     VERSION
ubuntu   Ready    master   7m14s   v1.16.2-k3s.1

root@ubuntu:~# kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   helm-install-traefik-szt4n                0/1     Completed   0          7m40s
kube-system   local-path-provisioner-5b8648d6f6-7fgm5   1/1     Running     1          7m40s
kube-system   coredns-66f496764-cjg7q                   1/1     Running     1          7m40s
kube-system   traefik-d869575c8-4gf95                   1/1     Running     1          4m39s
kube-system   svclb-traefik-vq8nb                       3/3     Running     0          32s
root@ubuntu:~# uname -a
Linux ubuntu 5.3.0-1008-raspi2 #9-Ubuntu SMP Fri Oct 18 13:26:35 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:        19.10
Codename:       eoan
root@ubuntu:~# k3s -version
k3s version v0.10.1 (7d650d32)

If I have time, will tried to directly install the latest version of K3s with the two cgroup option on a fresh Ubuntu 19.10 OS. At least I can run latest K3s on my Raspberry Pi4. But as of now, still don’t know if the issue relate to golang arm issue or other issue.

gm12367 on Oct 31, 2019