k3s: k3s won't start due to segv/Segmentation fault
We had built a cluster with 3 master nodes and a bunch of worker nodes. Over night 2 of the master died and didn’t came back up. Trying to start the k3s.service wie systemctl (Debian 10) is to no avail, the process gets killed immediately.
Dec 18 17:33:15 master-3 k3s[5645]: time="2019-12-18T17:33:15.329364828+01:00" level=info msg="Starting k3s v1.0.0 (18bd921c)"
Dec 18 17:33:15 master-3 k3s[5645]: time="2019-12-18T17:33:15.329884440+01:00" level=info msg="Cluster bootstrap already complete"
Dec 18 17:33:17 master-3 systemd[1]: k3s.service: Main process exited, code=killed, status=11/SEGV
Dec 18 17:33:17 master-3 systemd[1]: k3s.service: Failed with result 'signal'.
Dec 18 17:33:17 master-3 systemd[1]: Failed to start Lightweight Kubernetes.
Calling the k3s binary does basically the same.
I1218 17:36:26.952042 6205 interface.go:384] Looking for default routes with IPv4 addresses
I1218 17:36:26.952132 6205 interface.go:392] Default route transits interface "eth0"
I1218 17:36:26.952262 6205 interface.go:196] Interface eth0 is up
I1218 17:36:26.952327 6205 interface.go:244] Interface "eth0" has 3 addresses :[88.xx.xx.xx/32 2a01:xxx:xxx:352c::1/64 fe80::xx:xx:xx:b26b/64].
I1218 17:36:26.952359 6205 interface.go:211] Checking addr 88.xx.xx.xx/32.
I1218 17:36:26.952369 6205 interface.go:218] IP found 88.xx.xx.xx
I1218 17:36:26.952384 6205 interface.go:250] Found valid IPv4 address 88.xx.xx.xx for interface "eth0".
I1218 17:36:26.952392 6205 interface.go:398] Found active IP 88.xx.xx.xx
I1218 17:36:26.952418 6205 services.go:45] Setting service IP to "10.43.0.1" (read-write).
INFO[2019-12-18T17:36:26.952441448+01:00] Starting k3s v1.0.0 (18bd921c)
I1218 17:36:26.973940 6205 services.go:45] Setting service IP to "10.43.0.1" (read-write).
I1218 17:36:26.975450 6205 interface.go:384] Looking for default routes with IPv4 addresses
I1218 17:36:26.975470 6205 interface.go:392] Default route transits interface "eth0"
I1218 17:36:26.975540 6205 interface.go:196] Interface eth0 is up
I1218 17:36:26.975593 6205 interface.go:244] Interface "eth0" has 3 addresses :[88.xx.xx.xx/32 2a01:xxx:xxx:352c::1/64 fe80::xx:xx:xx:b26b/64].
I1218 17:36:26.975617 6205 interface.go:211] Checking addr 88.xx.xx.xx/32.
I1218 17:36:26.975627 6205 interface.go:218] IP found 88.xx.xx.xx/
I1218 17:36:26.975640 6205 interface.go:250] Found valid IPv4 address 88.xx.xx.xx for interface "eth0".
I1218 17:36:26.975656 6205 interface.go:398] Found active IP 88.xx.xx.xx
Segmentation fault
Cluster is build by running
curl -sfL https://get.k3s.io | K3S_TOKEN=<TOKEN> INSTALL_K3S_EXEC="server --no-deploy=traefik,local-storage,servicelb --flannel-backend=wireguard --cluster-init" sh -
We can’t get the cluster healthy again, because these two masters won’t start their k3s again. Any idea how to fix?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 18 (1 by maintainers)
The issue still exists on Debian 10 amd64 with
v1.17.4+k3s1in a multi server environment with embedded dsqlite DB.