k3s: Panic with 1.19.5+k3s1 on rpi-4 server

Environmental Info: K3s Version:

pi@pi-41:~ $ k3s -v
k3s version v1.19.5+k3s1 (b11612e2)

Node(s) CPU architecture, OS, and Version:

pi@pi-41:~ $ uname -a
Linux pi-41 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux

Cluster Configuration: 4 nodes cluster running on rpi-4s (2GB): 1 server + 3 workers

Describe the bug: After updating the cluster from v1.19.4+k3s2 to v1.19.5+k3s1, the server starts panicking with unexpected fault address on server’s startup.

From the logs (see below) it looks like the crash happens after the server reaches “CronJob Manager”. In fact, the only workload I run on the server for the last couple months was this cronjob.

Steps To Reproduce:

Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

An extract from the stack trace

Dec 14 23:30:02 pi-41 k3s[525]: I1214 23:30:02.891889     525 shared_informer.go:240] Waiting for caches to sync for disruption
Dec 14 23:30:02 pi-41 k3s[525]: I1214 23:30:02.984144     525 controllermanager.go:549] Started "cronjob"
Dec 14 23:30:02 pi-41 k3s[525]: I1214 23:30:02.984415     525 cronjob_controller.go:96] Starting CronJob Manager
Dec 14 23:30:03 pi-41 k3s[525]: unexpected fault address 0x123ee0d0
Dec 14 23:30:03 pi-41 k3s[525]: fatal error: fault
Dec 14 23:30:03 pi-41 k3s[525]: [signal SIGSEGV: segmentation violation code=0x2 addr=0x123ee0d0 pc=0x123ee0d0]
Dec 14 23:30:03 pi-41 k3s[525]: goroutine 43396 [running]:
Dec 14 23:30:03 pi-41 k3s[525]: runtime.throw(0x3372b14, 0x5)
Dec 14 23:30:03 pi-41 k3s[525]:         /usr/local/go/src/runtime/panic.go:1116 +0x5c fp=0x11b60b90 sp=0x11b60b7c pc=0x46154
Dec 14 23:30:03 pi-41 k3s[525]: runtime.sigpanic()
Dec 14 23:30:03 pi-41 k3s[525]:         /usr/local/go/src/runtime/signal_unix.go:749 +0x310 fp=0x11b60ba8 sp=0x11b60b90 pc=0x5d688
Dec 14 23:30:03 pi-41 k3s[525]: runtime: unexpected return pc for github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.getFinishedStatus called from 0x0
Dec 14 23:30:03 pi-41 k3s[525]: stack: frame={sp:0x11b60bac, fp:0x11b60c5c} stack=[0x11b5e000,0x11b62000)
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b2c:  00047eb0 <runtime.gwrite+52>  057e8698  0336ead4  00000001
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b3c:  00000001  00000000  00048748 <runtime.printstring+96>  00000002
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b4c:  0336ead4  00000001  00000001  00000000
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b5c:  00000000  000462f4 <runtime.fatalthrow+68>  00046154 <runtime.throw+92>  11b60b6c
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b6c:  000748fc <runtime.fatalthrow.func1+0>  0e767ea0  00046154 <runtime.throw+92>  11b60b7c
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b7c:  0005d688 <runtime.sigpanic+784>  11b60b84  00074890 <runtime.throw.func1+0>  03372b14
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b8c:  00000005  0207c614 <github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.getFinishedStatus+88>  03372b14  00000005
Dec 14 23:30:03 pi-41 k3s[525]: 11b60b9c:  00000001  123ee0d0  0e767ea0  0207c614 <github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.getFinishedStatus+88>
Dec 14 23:30:03 pi-41 k3s[525]: 11b60bac: <00000000  00000001  a6915110  00000000
Dec 14 23:30:03 pi-41 k3s[525]: 11b60bbc:  00000064  123ee0d0  00085684 <sync.(*Pool).pin+48>  00000065
Dec 14 23:30:03 pi-41 k3s[525]: 11b60bcc:  00000016  b6ee93ec  00085684 <sync.(*Pool).pin+48>  0008524c <sync.(*Pool).Put+40>
Dec 14 23:30:03 pi-41 k3s[525]: 11b60bdc:  05577fe4  00000004  00000700  00000001
Dec 14 23:30:03 pi-41 k3s[525]: 11b60bec:  00117e10 <fmt.(*pp).free+160>  00022e08 <runtime.mapassign_faststr+72>  18d901b0  5b731b39
Dec 14 23:30:03 pi-41 k3s[525]: 11b60bfc:  00000024  0f19b06c  02079030 <github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.syncOne+412>  11b60c40
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c0c:  5b731b39  0f19b06c  15142300  0000000f
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c1c:  0d08ac80  00000016  0f19b06c  00000016
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c2c:  0d08ac80  11b611bc  0207ce60 <github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.(*Controller).syncAll.func4+208>  11b61210
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c3c:  00000000  00000000  00000000  139ea950
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c4c:  0000000e  00000000  00000000  139ea960
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c5c: >00000007  09fedbc0  0000003e  18cbfd70
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c6c:  00000024  139ea967  00000007  00000000
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c7c:  00000000  00000000  00000000  d72a7c3f
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c8c:  0000000e  057d12a8  00000000  00000000
Dec 14 23:30:03 pi-41 k3s[525]: 11b60c9c:  00000000  0c4a2080  00000000  00000000
Dec 14 23:30:03 pi-41 k3s[525]: 11b60cac:  00000000  00000000  00000000  00000000
Dec 14 23:30:03 pi-41 k3s[525]: 11b60cbc:  00000000  00000000  130f8550  00000002
Dec 14 23:30:03 pi-41 k3s[525]: 11b60ccc:  00000002  139ea9d0  0000000c  00000000
Dec 14 23:30:03 pi-41 k3s[525]: github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.getFinishedStatus(0x9fedbc0, 0x3e, 0x18cbfd70, 0x24)
Dec 14 23:30:03 pi-41 k3s[525]:         /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob/utils.go:175 +0x58 fp=0x11b60c5c sp=0x11b60bac pc=0x207c614
Dec 14 23:30:03 pi-41 k3s[525]: created by github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob.(*Controller).Run
Dec 14 23:30:03 pi-41 k3s[525]:         /go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/controller/cronjob/cronjob_controller.go:98 +0xf8
Dec 14 23:30:03 pi-41 k3s[525]: goroutine 1 [chan receive]:
Dec 14 23:30:03 pi-41 k3s[525]: github.com/rancher/k3s/pkg/agent.run(0x3a58288, 0x7766340, 0x139e49a0, 0x6a, 0x0, 0x0, 0x0, 0x0, 0xcfaf320, 0x16, ...)
Dec 14 23:30:03 pi-41 k3s[525]:         /go/src/github.com/rancher/k3s/pkg/agent/run.go:119 +0x1c0
Dec 14 23:30:03 pi-41 k3s[525]: github.com/rancher/k3s/pkg/agent.Run(0x3a58288, 0x7766340, 0x139e49a0, 0x6a, 0x0, 0x0, 0x0, 0x0, 0xcfaf320, 0x16, ...)
Dec 14 23:30:03 pi-41 k3s[525]:         /go/src/github.com/rancher/k3s/pkg/agent/run.go:170 +0x2f4
Dec 14 23:30:03 pi-41 k3s[525]: github.com/rancher/k3s/pkg/cli/server.run(0x774c370, 0x57d1ed0, 0x48, 0x9f0f34)
Dec 14 23:30:03 pi-41 k3s[525]:         /go/src/github.com/rancher/k3s/pkg/cli/server/server.go:283 +0x12c0
Dec 14 23:30:03 pi-41 k3s[525]: github.com/rancher/k3s/pkg/cli/server.Run(0x774c370, 0x0, 0x774c370)

Full logs pi-41-k3s-daemon.1.log.gz

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 4
Comments: 24 (17 by maintainers)

Commits related to this issue

Downgrade server to work around cronjob bug, disable CC to work around startup bug https://github.com/k3s-io/k3s/issues/2704 https://github.com/k3s-io/k3s/issues/2425 — committed to parente/homelab by parente 3 years ago

Most upvoted comments

See if this is fixed on v1.20.2+k3s1

Running v1.20.2+k3s1 on the server node (Raspbian GNU/Linux 10 (buster), 5.4.83-v7l+, armv7l) for that last 24h, and it seems to work fine 🎉

narqo on Jan 16, 2021