k3s: k3s won't start due to segv/Segmentation fault

We had built a cluster with 3 master nodes and a bunch of worker nodes. Over night 2 of the master died and didn’t came back up. Trying to start the k3s.service wie systemctl (Debian 10) is to no avail, the process gets killed immediately.

Dec 18 17:33:15 master-3 k3s[5645]: time="2019-12-18T17:33:15.329364828+01:00" level=info msg="Starting k3s v1.0.0 (18bd921c)"
Dec 18 17:33:15 master-3 k3s[5645]: time="2019-12-18T17:33:15.329884440+01:00" level=info msg="Cluster bootstrap already complete"
Dec 18 17:33:17 master-3 systemd[1]: k3s.service: Main process exited, code=killed, status=11/SEGV
Dec 18 17:33:17 master-3 systemd[1]: k3s.service: Failed with result 'signal'.
Dec 18 17:33:17 master-3 systemd[1]: Failed to start Lightweight Kubernetes.

Calling the k3s binary does basically the same.

I1218 17:36:26.952042    6205 interface.go:384] Looking for default routes with IPv4 addresses
I1218 17:36:26.952132    6205 interface.go:392] Default route transits interface "eth0"
I1218 17:36:26.952262    6205 interface.go:196] Interface eth0 is up
I1218 17:36:26.952327    6205 interface.go:244] Interface "eth0" has 3 addresses :[88.xx.xx.xx/32 2a01:xxx:xxx:352c::1/64 fe80::xx:xx:xx:b26b/64].
I1218 17:36:26.952359    6205 interface.go:211] Checking addr  88.xx.xx.xx/32.
I1218 17:36:26.952369    6205 interface.go:218] IP found 88.xx.xx.xx
I1218 17:36:26.952384    6205 interface.go:250] Found valid IPv4 address 88.xx.xx.xx for interface "eth0".
I1218 17:36:26.952392    6205 interface.go:398] Found active IP 88.xx.xx.xx
I1218 17:36:26.952418    6205 services.go:45] Setting service IP to "10.43.0.1" (read-write).
INFO[2019-12-18T17:36:26.952441448+01:00] Starting k3s v1.0.0 (18bd921c)
I1218 17:36:26.973940    6205 services.go:45] Setting service IP to "10.43.0.1" (read-write).
I1218 17:36:26.975450    6205 interface.go:384] Looking for default routes with IPv4 addresses
I1218 17:36:26.975470    6205 interface.go:392] Default route transits interface "eth0"
I1218 17:36:26.975540    6205 interface.go:196] Interface eth0 is up
I1218 17:36:26.975593    6205 interface.go:244] Interface "eth0" has 3 addresses :[88.xx.xx.xx/32 2a01:xxx:xxx:352c::1/64 fe80::xx:xx:xx:b26b/64].
I1218 17:36:26.975617    6205 interface.go:211] Checking addr  88.xx.xx.xx/32.
I1218 17:36:26.975627    6205 interface.go:218] IP found 88.xx.xx.xx/
I1218 17:36:26.975640    6205 interface.go:250] Found valid IPv4 address 88.xx.xx.xx for interface "eth0".
I1218 17:36:26.975656    6205 interface.go:398] Found active IP 88.xx.xx.xx
Segmentation fault

Cluster is build by running curl -sfL https://get.k3s.io | K3S_TOKEN=<TOKEN> INSTALL_K3S_EXEC="server --no-deploy=traefik,local-storage,servicelb --flannel-backend=wireguard --cluster-init" sh -

We can’t get the cluster healthy again, because these two masters won’t start their k3s again. Any idea how to fix?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 3
  • Comments: 18 (1 by maintainers)

Most upvoted comments

The issue still exists on Debian 10 amd64 with v1.17.4+k3s1 in a multi server environment with embedded dsqlite DB.