go: runtime: morestack_noctxt missing SPWRITE, causes "traceback stuck" assert

What version of Go are you using (`go version`)?

$ go version
go version go1.18.5 linux/arm64

Does this issue reproduce with the latest release?

yes, repros on go1.19.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/lizf/.cache/go-build"
GOENV="/home/lizf/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/lizf/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/lizf/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm64"
GOVCS=""
GOVERSION="go1.18.5"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3053713255=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run continuous profiling via scraping /debug/pprof endpoints including mutex and blocking profiles, while performing a workload consuming from Kafka using Sarama and Klauspost Zstd libraries.

  runtime.SetBlockProfileRate(100000)
  runtime.SetMutexProfileFraction(100)

What did you expect to see?

No crashes.

What did you see instead?

The following crashes implicating unexpected return pc for runtime.sigtramp and traceback stuck (assert added in https://go-review.googlesource.com/c/go/+/400575/ by @cherrymui / #52116 which was backported to go1.18.5):

runtime: traceback stuck. pc=0x12eb5 sp=0x4019365840
stack: frame={sp:0x4019365840, fp:0x4019365840} stack=[0x4019364000,0x4019366000)
0x0000004019365740:  0x0000000000ba9494 <github.com/Shopify/sarama.releaseCrc32Field+0x0000000000000034>  0x00000040193657b8 
0x0000004019365750:  0x0000000000bcd8e0 <github.com/Shopify/sarama.(*realDecoder).pop+0x0000000000000060>  0x00000040193657c8 
0x0000004019365760:  0x0000000000000000  0x0000000000000000 
0x0000004019365770:  0x0000000000000700  0x0000000001068260 
0x0000004019365780:  0x00000000010ab360  0x0000000000005701 
0x0000004019365790:  0x0000000000000000  0x0000000000000000 
0x00000040193657a0:  0x0000000000000000  0x0000004000601c00 
0x00000040193657b0:  0x000000401c3c7100  0x00000040193657f8 
0x00000040193657c0:  0x0000000000bd070c <github.com/Shopify/sarama.(*RecordBatch).decode+0x000000000000037c>  0x000000400ff4e0f0 
0x00000040193657d0:  0x0000000000bd064c <github.com/Shopify/sarama.(*RecordBatch).decode+0x00000000000002bc>  0x0000004026c7805f 
0x00000040193657e0:  0x0000000000107de8 <strings.HasSuffix+0x0000000000000058>  0x0000000000000000 
0x00000040193657f0:  0x0000000000000000  0x00000040193658c8 
0x0000004019365800:  0x0000000000bd0eac <github.com/Shopify/sarama.(*Records).decode+0x000000000000009c>  0x0000004017c948c0 
0x0000004019365810:  0x0000004026d6cf92  0x0000000000012eb5 <internal/bytealg.IndexByteString+0x0000000000000005> 
0x0000004019365820:  0x0000000000012eb5 <internal/bytealg.IndexByteString+0x0000000000000005>  0x0000004019365898 
0x0000004019365830:  0x00012ee60001ce60  0x0000000000012eb5 <internal/bytealg.IndexByteString+0x0000000000000005> 
0x0000004019365840: >0x0000000000012eb5 <internal/bytealg.IndexByteString+0x0000000000000005>  0x00000000000000d7 
0x0000004019365850:  0x0000000000000000  0x0000004019365800 
0x0000004019365860:  0x0000000000bd0474 <github.com/Shopify/sarama.(*RecordBatch).decode+0x00000000000000e4>  0x00000040193658b8 
0x0000004019365870:  0x0000000000000000  0x0000000000000000 
0x0000004019365880:  0x0000000000000000  0x0000000000000000 
0x0000004019365890:  0x0000000000000000  0x0000004026d6cf92 
0x00000040193658a0:  0x000000400ff4e0f0  0x0000000000000000 
0x00000040193658b0:  0x0000000000000000  0x0000000000bd09c0 <github.com/Shopify/sarama.(*RecordBatch).decode.func1+0x0000000000000000> 
0x00000040193658c0:  0x000000400ff4e0f0  0x0000004019365918 
0x00000040193658d0:  0x0000000000bb6f1c <github.com/Shopify/sarama.(*FetchResponseBlock).decode+0x000000000000033c>  0x0000004012612c80 
0x00000040193658e0:  0x000000000154bfa8  0x0000004017c948c0 
0x00000040193658f0:  0x0000000000bb6f0c <github.com/Shopify/sarama.(*FetchResponseBlock).decode+0x000000000000032c>  0x0000000000000018 
0x0000004019365900:  0x00000000010d87c0  0x0000004000048301 
0x0000004019365910:  0x0000004017c94a40  0x00000040193659c8 
0x0000004019365920:  0x0000000000bb78e8 <github.com/Shopify/sarama.(*FetchResponse).decode+0x00000000000002a8>  0x0000004009b9fc38 
0x0000004019365930:  0x000000000154bfa8  0x0000004017c948c0 
fatal error: traceback stuck

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0xffff985dc5c0
stack: frame={sp:0x400eacacd0, fp:0x400eacada0} stack=[0x400eac4000,0x400eacc000)
0x000000400eacabd0:  0x000000400eacac28  0x02c595e603246a9e 
0x000000400eacabe0:  0x0000000002c595e5  0x000000400eaba000 
0x000000400eacabf0:  0x0000000000000000  0x000000400eac9fe0 
0x000000400eacac00:  0x0000004013a80000  0x000000400eacada0 
0x000000400eacac10:  0x000000400eacae20  0x000000400eacac98 
0x000000400eacac20:  0x000000000007d58c <runtime.sigtrampgo+0x000000000000001c>  0x000000400000001b 
0x000000400eacac30:  0x0000004000601c00  0x000000400eacac58 
0x000000400eacac40:  0x0000004000603040  0x02bcec7502c540fb 
0x000000400eacac50:  0x00246aa103246aa0  0x0000000000000000 
0x000000400eacac60:  0x0000000000000000  0x0000000000000000 
0x000000400eacac70:  0x0000000000000000  0x0000000000000000 
0x000000400eacac80:  0x0000004000603040  0x000000400eacada0 
0x000000400eacac90:  0x000000400eacae20  0x000000400eacacc8 
0x000000400eacaca0:  0x000000000007cea4 <runtime.sigtramp+0x0000000000000084>  0x000000000000001b 
0x000000400eacacb0:  0x000000400eacada0  0x000000400eacae20 
0x000000400eacacc0:  0x02c5956e03246a9e  0x000000400eacbff0 
0x000000400eacacd0: <0x0000ffff985dc5c0  0x000000400000001b 
0x000000400eacace0:  0x000000400eacada0  0x000000400eacae20 
0x000000400eacacf0:  0x000000000005abba <runtime.selparkcommit+0x000000000000000a>  0x1265726f74537070 
0x000000400eacad00:  0x021210eed9a50809  0x623824220118000a 
0x000000400eacad10:  0x302d333733663636  0x0000000000000015 
0x000000400eacad20:  0x0000000000000071  0x000000400b753e98 
0x000000400eacad30:  0x000000000153a0c0  0x0000004000603040 
0x000000400eacad40:  0x000000400eacacc8  0x3e66376972bea4d0 
0x000000400eacad50:  0xbebbbd41c5d26bf1  0x3f11566aaf25de2c 
0x000000400eacad60:  0xbf66c16c16bebd93  0x3fc5555555555555 
0x000000400eacad70:  0x3fcde3175e2d734a  0x3ffc439d143a5197 
0x000000400eacad80:  0xbfd1a21a8e7d2cb5  0x0000000000000000 
0x000000400eacad90:  0x0404040202020303  0x0000000000000000 
0x000000400eacada0: >0x000000000000001b  0x00000000fffffffe 
0x000000400eacadb0:  0x0000000000000466  0x0000000000000000 
0x000000400eacadc0:  0x0000000000000000  0x0000000000000000 
0x000000400eacadd0:  0x0000000000000000  0x0000000000000000 
0x000000400eacade0:  0x0000000000000000  0x0000000000000000 
0x000000400eacadf0:  0x0000000000000000  0x0000000000000000 
0x000000400eacae00:  0x0000000000000000  0x0000000000000000 
0x000000400eacae10:  0x0000000000000000  0x0000000000000000 
0x000000400eacae20:  0x0000000000000000  0x0000000000000000 
0x000000400eacae30:  0x000000400eac4000  0x0000004000000000 
0x000000400eacae40:  0x0000000000008000  0x0000000000000000 
0x000000400eacae50:  0x0000000000000000  0x0000000000000000 
0x000000400eacae60:  0x0000000000000000  0x0000000000000000 
0x000000400eacae70:  0x00002a1c49103b0c  0x02bcec6f02b9baa6 
0x000000400eacae80:  0x02b9baaf03246a9f  0x02c5956e02bcec75 
0x000000400eacae90:  0x0000000000000000  0x0000004013a805d0 
runtime.throw({0x1262952?, 0x1fae6c0?})
	/usr/local/go/src/runtime/panic.go:992 +0x50
runtime.gentraceback(0x203ad80?, 0x400eac9a90?, 0x400eaca968?, 0x4000603040, 0x0, 0x400eaca968, 0x40, 0x0, 0x0?, 0x6)
	/usr/local/go/src/runtime/traceback.go:488 +0xd98
runtime.sigprof(0x79650, 0x400eacac08?, 0x4013a80010?, 0x4000603040, 0x4000601c00)
	/usr/local/go/src/runtime/proc.go:4507 +0xf0
runtime.sighandler(0x1b?, 0x4000601c00?, 0x400eacac58?, 0x4000603040?)
	/usr/local/go/src/runtime/signal_unix.go:613 +0x514
runtime.sigtrampgo(0x1b, 0x400eacada0, 0x400eacae20)
	/usr/local/go/src/runtime/signal_unix.go:477 +0x164
runtime.sigtrampgo(0x1b, 0x400eacada0, 0x400eacae20)
	<autogenerated>:1 +0x1c
runtime: unexpected return pc for runtime.sigtramp called from 0xffff985dc5c0
stack: frame={sp:0x400eacacd0, fp:0x400eacada0} stack=[0x400eac4000,0x400eacc000)
0x000000400eacabd0:  0x000000400eacac28  0x02c595e603246a9e 
0x000000400eacabe0:  0x0000000002c595e5  0x000000400eaba000 
0x000000400eacabf0:  0x0000000000000000  0x000000400eac9fe0 
0x000000400eacac00:  0x0000004013a80000  0x000000400eacada0 
0x000000400eacac10:  0x000000400eacae20  0x000000400eacac98 
0x000000400eacac20:  0x000000000007d58c <runtime.sigtrampgo+0x000000000000001c>  0x000000400000001b 
0x000000400eacac30:  0x0000004000601c00  0x000000400eacac58 
0x000000400eacac40:  0x0000004000603040  0x02bcec7502c540fb 
0x000000400eacac50:  0x00246aa103246aa0  0x0000000000000000 
0x000000400eacac60:  0x0000000000000000  0x0000000000000000 
0x000000400eacac70:  0x0000000000000000  0x0000000000000000 
0x000000400eacac80:  0x0000004000603040  0x000000400eacada0 
0x000000400eacac90:  0x000000400eacae20  0x000000400eacacc8 
0x000000400eacaca0:  0x000000000007cea4 <runtime.sigtramp+0x0000000000000084>  0x000000000000001b 
0x000000400eacacb0:  0x000000400eacada0  0x000000400eacae20 
0x000000400eacacc0:  0x02c5956e03246a9e  0x000000400eacbff0 
0x000000400eacacd0: <0x0000ffff985dc5c0  0x000000400000001b 
0x000000400eacace0:  0x000000400eacada0  0x000000400eacae20 
0x000000400eacacf0:  0x000000000005abba <runtime.selparkcommit+0x000000000000000a>  0x1265726f74537070 
0x000000400eacad00:  0x021210eed9a50809  0x623824220118000a 
0x000000400eacad10:  0x302d333733663636  0x0000000000000015 
0x000000400eacad20:  0x0000000000000071  0x000000400b753e98 
0x000000400eacad30:  0x000000000153a0c0  0x0000004000603040 
0x000000400eacad40:  0x000000400eacacc8  0x3e66376972bea4d0 
0x000000400eacad50:  0xbebbbd41c5d26bf1  0x3f11566aaf25de2c 
0x000000400eacad60:  0xbf66c16c16bebd93  0x3fc5555555555555 
0x000000400eacad70:  0x3fcde3175e2d734a  0x3ffc439d143a5197 
0x000000400eacad80:  0xbfd1a21a8e7d2cb5  0x0000000000000000 
0x000000400eacad90:  0x0404040202020303  0x0000000000000000 
0x000000400eacada0: >0x000000000000001b  0x00000000fffffffe 
0x000000400eacadb0:  0x0000000000000466  0x0000000000000000 
0x000000400eacadc0:  0x0000000000000000  0x0000000000000000 
0x000000400eacadd0:  0x0000000000000000  0x0000000000000000 
0x000000400eacade0:  0x0000000000000000  0x0000000000000000 
0x000000400eacadf0:  0x0000000000000000  0x0000000000000000 
0x000000400eacae00:  0x0000000000000000  0x0000000000000000 
0x000000400eacae10:  0x0000000000000000  0x0000000000000000 
0x000000400eacae20:  0x0000000000000000  0x0000000000000000 
0x000000400eacae30:  0x000000400eac4000  0x0000004000000000 
0x000000400eacae40:  0x0000000000008000  0x0000000000000000 
0x000000400eacae50:  0x0000000000000000  0x0000000000000000 
0x000000400eacae60:  0x0000000000000000  0x0000000000000000 
0x000000400eacae70:  0x00002a1c49103b0c  0x02bcec6f02b9baa6 
0x000000400eacae80:  0x02b9baaf03246a9f  0x02c5956e02bcec75 
0x000000400eacae90:  0x0000000000000000  0x0000004013a805d0 
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_linux_arm64.s:489 +0x84

goroutine 89 [running]:

runtime: traceback stuck. pc=0x3ff93 sp=0x40048952c0
stack: frame={sp:0x40048952c0, fp:0x40048952c0} stack=[0x4004894000,0x4004896000)
0x00000040048951c0:  0x0000000000000040  0x00000040048951e8 
0x00000040048951d0:  0x00000000005019a4 <github.com/klauspost/compress/zstd.(*sequenceDec).init+0x0000000000000054>  0x0000004004895310 
0x00000040048951e0:  0x00000040008c6000  0x0000004004895228 
0x00000040048951f0:  0x0000000000501b9c <github.com/klauspost/compress/zstd.(*sequenceDecs).initialize+0x000000000000016c>  0x0000004004895338 
0x0000004004895200:  0x00000000004e2b84 <github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed+0x0000000000000164>  0x0000004000530058 
0x0000004004895210:  0x00000040002e22a0  0x00000000006c0001 <text/template/parse.(*endNode).Copy+0x0000000000000001> 
0x0000004004895220:  0x0000000f00800005  0x000001fd00280903 
0x0000004004895230:  0x00000000004e2ebc <github.com/klauspost/compress/zstd.(*blockDec).prepareSequences+0x000000000000025c>  0x000000000003ff93 <runtime.goroutineProfileWithLabels.func4.1+0x0000000000000003> 
0x0000004004895240:  0x00000000000ee1a4 <os.UserHomeDir+0x00000000000000c4>  0x0000000000000811 
0x0000004004895250:  0x0000000000000015  0x0000000000020000 <runtime.advanceEvacuationMark+0x0000000000000000> 
0x0000004004895260:  0x00000000006c0001 <text/template/parse.(*endNode).Copy+0x0000000000000001>  0x0000000000000000 
0x0000004004895270:  0x0000000000000006  0x0000000f00800005 
0x0000004004895280:  0x000001fd00280903  0x0000000000000000 
0x0000004004895290:  0x000000400720ea00  0x0000004004895338 
0x00000040048952a0:  0x00000000004e2a78 <github.com/klauspost/compress/zstd.(*blockDec).decodeCompressed+0x0000000000000058>  0x0000004000530058 
0x00000040048952b0:  0x00000040002e22a0  0x00000000000ee1a4 <os.UserHomeDir+0x00000000000000c4> 
0x00000040048952c0: >0x000000000003ff93 <runtime.goroutineProfileWithLabels.func4.1+0x0000000000000003>  0x0000000000020000 <runtime.advanceEvacuationMark+0x0000000000000000> 
0x00000040048952d0:  0x0000004009f42000  0x00000040002e22a0 
0x00000040048952e0:  0x0000000000001a81  0x0000004009f42000 
0x00000040048952f0:  0x0000004005b17500  0x000000400720ea00 
0x0000004004895300:  0x0000004004a28a00  0x00000040002e22a0 
0x0000004004895310:  0x0000004003bcf000  0x000000400885150c 
0x0000004004895320:  0x000000000000467b  0x000000000002ba01 <runtime.gcStart+0x0000000000000701> 
0x0000004004895330:  0x000000000000002e  0x0000004004895398 
0x0000004004895340:  0x00000000004e1974 <github.com/klauspost/compress/zstd.(*blockDec).decodeBuf+0x0000000000000284>  0x0000004000530058 
0x0000004004895350:  0x0000000000000000  0x0000000000000000 
0x0000004004895360:  0x0000000000000000  0x0000004000530050 
0x0000004004895370:  0x0000000000000000  0x0000000000000000 
0x0000004004895380:  0x0000000000000000  0x0000000000020000 <runtime.advanceEvacuationMark+0x0000000000000000> 
0x0000004004895390:  0x0000000000000000  0x0000004004895458 
0x00000040048953a0:  0x00000000004fd2cc <github.com/klauspost/compress/zstd.(*frameDec).runDecoder+0x000000000000016c>  0x000000400058c000 
0x00000040048953b0:  0x0000004000530050  0x0000000000000000 
fatal error: traceback stuck
goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0xffffabfd25c0
stack: frame={sp:0x400000acd0, fp:0x400000ada0} stack=[0x4000004000,0x400000c000)
0x000000400000abd0:  0x000000400000ac28  0x0000000000000000 
0x000000400000abe0:  0x0000000000000000  0x0000004000002000 
0x000000400000abf0:  0x0000000000000000  0x0000000000000000 
0x000000400000ac00:  0x0000000000000000  0x000000400000ada0 
0x000000400000ac10:  0x000000400000ae20  0x000000400000ac98 
0x000000400000ac20:  0x000000000007d58c <runtime.sigtrampgo+0x000000000000001c>  0x000000400000001b 
0x000000400000ac30:  0x0000000002046680  0x000000400000ac58 
0x000000400000ac40:  0x0000004000521d40  0x0000000000000000 
0x000000400000ac50:  0x0000000000000000  0x0000000000000000 
0x000000400000ac60:  0x0000000000000000  0x0000000000000000 
0x000000400000ac70:  0x0000000000000000  0x0000000000000000 
0x000000400000ac80:  0x0000004000521d40  0x000000400000ada0 
0x000000400000ac90:  0x000000400000ae20  0x000000400000acc8 
0x000000400000aca0:  0x000000000007cea4 <runtime.sigtramp+0x0000000000000084>  0x000000000000001b 
0x000000400000acb0:  0x000000400000ada0  0x000000400000ae20 
0x000000400000acc0:  0x0000000000000000  0x000000400000bff0 
0x000000400000acd0: <0x0000ffffabfd25c0  0x000000000000001b 
0x000000400000ace0:  0x000000400000ada0  0x000000400000ae20 
0x000000400000acf0:  0x0000000000000009  0x0000000000000015 
0x000000400000ad00:  0x0000000000000000  0x00000040008c9fab 
0x000000400000ad10:  0x000000000001ff93 <runtime.evacuate+0x00000000000004b3>  0x0000000000000009 
0x000000400000ad20:  0x000000000000032d  0x00000040048953d0 
0x000000400000ad30:  0x0000000002078100  0x0000004000521d40 
0x000000400000ad40:  0x000000400000acc8  0x3e66376972bea4d0 
0x000000400000ad50:  0xbebbbd41c5d26bf1  0x3f11566aaf25de2c 
0x000000400000ad60:  0xbf66c16c16bebd93  0x3fc5555555555555 
0x000000400000ad70:  0x3fb8fcd2e18d7b62  0x3ffe7032d1e7284a 
0x000000400000ad80:  0xbfbab6204490a1e0  0x0000000000000000 
0x000000400000ad90:  0x0000000000000000  0x0000000000000000 
0x000000400000ada0: >0x000000000000001b  0x00000000fffffffe 
0x000000400000adb0:  0x0000000000004d16  0x0000000000000000 
0x000000400000adc0:  0x0000000000000000  0x0000000000000000 
0x000000400000add0:  0x0000000000000000  0x0000000000000000 
0x000000400000ade0:  0x0000000000000000  0x0000000000000000 
0x000000400000adf0:  0x0000000000000000  0x0000000000000000 
0x000000400000ae00:  0x0000000000000000  0x0000000000000000 
0x000000400000ae10:  0x0000000000000000  0x0000000000000000 
0x000000400000ae20:  0x0000000000000000  0x0000000000000000 
0x000000400000ae30:  0x0000004000004000  0x0000000000000000 
0x000000400000ae40:  0x0000000000008000  0x0000000000000000 
0x000000400000ae50:  0x0000000000000000  0x0000000000000000 
0x000000400000ae60:  0x0000000000000000  0x0000000000000000 
0x000000400000ae70:  0x0000000000000000  0x0000000000000000 
0x000000400000ae80:  0x0000000000000000  0x0000000000000000 
0x000000400000ae90:  0x0000000000000000  0x0000000000000000 
runtime.throw({0x1275238?, 0x1fbe760?})
    /usr/local/go/src/runtime/panic.go:992 +0x50
runtime.gentraceback(0x204afc0?, 0x0?, 0x400000a968?, 0x4000521d40, 0x0, 0x400000a968, 0x40, 0x0, 0x0?, 0x6)
    /usr/local/go/src/runtime/traceback.go:488 +0xd98
runtime.sigprof(0x79650, 0x400000ac08?, 0x0?, 0x4000521d40, 0x2046680)
    /usr/local/go/src/runtime/proc.go:4507 +0xf0
runtime.sighandler(0x1b?, 0x2046680?, 0x400000ac58?, 0x4000521d40?)
    /usr/local/go/src/runtime/signal_unix.go:613 +0x514
runtime.sigtrampgo(0x1b, 0x400000ada0, 0x400000ae20)
    /usr/local/go/src/runtime/signal_unix.go:477 +0x164
runtime.sigtrampgo(0x1b, 0x400000ada0, 0x400000ae20)
    <autogenerated>:1 +0x1c
runtime: unexpected return pc for runtime.sigtramp called from 0xffffabfd25c0
stack: frame={sp:0x400000acd0, fp:0x400000ada0} stack=[0x4000004000,0x400000c000)
0x000000400000abd0:  0x000000400000ac28  0x0000000000000000 
0x000000400000abe0:  0x0000000000000000  0x0000004000002000 
0x000000400000abf0:  0x0000000000000000  0x0000000000000000 
0x000000400000ac00:  0x0000000000000000  0x000000400000ada0 
0x000000400000ac10:  0x000000400000ae20  0x000000400000ac98 
0x000000400000ac20:  0x000000000007d58c <runtime.sigtrampgo+0x000000000000001c>  0x000000400000001b 
0x000000400000ac30:  0x0000000002046680  0x000000400000ac58 
0x000000400000ac40:  0x0000004000521d40  0x0000000000000000 
0x000000400000ac50:  0x0000000000000000  0x0000000000000000 
0x000000400000ac60:  0x0000000000000000  0x0000000000000000 
0x000000400000ac70:  0x0000000000000000  0x0000000000000000 
0x000000400000ac80:  0x0000004000521d40  0x000000400000ada0 
0x000000400000ac90:  0x000000400000ae20  0x000000400000acc8 
0x000000400000aca0:  0x000000000007cea4 <runtime.sigtramp+0x0000000000000084>  0x000000000000001b 
0x000000400000acb0:  0x000000400000ada0  0x000000400000ae20 
0x000000400000acc0:  0x0000000000000000  0x000000400000bff0 
0x000000400000acd0: <0x0000ffffabfd25c0  0x000000000000001b 
0x000000400000ace0:  0x000000400000ada0  0x000000400000ae20 
0x000000400000acf0:  0x0000000000000009  0x0000000000000015 
0x000000400000ad00:  0x0000000000000000  0x00000040008c9fab 
0x000000400000ad10:  0x000000000001ff93 <runtime.evacuate+0x00000000000004b3>  0x0000000000000009 
0x000000400000ad20:  0x000000000000032d  0x00000040048953d0 
0x000000400000ad30:  0x0000000002078100  0x0000004000521d40 
0x000000400000ad40:  0x000000400000acc8  0x3e66376972bea4d0 
0x000000400000ad50:  0xbebbbd41c5d26bf1  0x3f11566aaf25de2c 
0x000000400000ad60:  0xbf66c16c16bebd93  0x3fc5555555555555 
0x000000400000ad70:  0x3fb8fcd2e18d7b62  0x3ffe7032d1e7284a 
0x000000400000ad80:  0xbfbab6204490a1e0  0x0000000000000000 
0x000000400000ad90:  0x0000000000000000  0x0000000000000000 
0x000000400000ada0: >0x000000000000001b  0x00000000fffffffe 
0x000000400000adb0:  0x0000000000004d16  0x0000000000000000 
0x000000400000adc0:  0x0000000000000000  0x0000000000000000 
0x000000400000add0:  0x0000000000000000  0x0000000000000000 
0x000000400000ade0:  0x0000000000000000  0x0000000000000000 
0x000000400000adf0:  0x0000000000000000  0x0000000000000000 
0x000000400000ae00:  0x0000000000000000  0x0000000000000000 
0x000000400000ae10:  0x0000000000000000  0x0000000000000000 
0x000000400000ae20:  0x0000000000000000  0x0000000000000000 
0x000000400000ae30:  0x0000004000004000  0x0000000000000000 
0x000000400000ae40:  0x0000000000008000  0x0000000000000000 
0x000000400000ae50:  0x0000000000000000  0x0000000000000000 
0x000000400000ae60:  0x0000000000000000  0x0000000000000000 
0x000000400000ae70:  0x0000000000000000  0x0000000000000000 
0x000000400000ae80:  0x0000000000000000  0x0000000000000000 
0x000000400000ae90:  0x0000000000000000  0x0000000000000000 
runtime.sigtramp()
    /usr/local/go/src/runtime/sys_linux_arm64.s:489 +0x84
goroutine 197 [running]:

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 68 (25 by maintainers)

Commits related to this issue

[release-branch.go1.18] runtime: mark morestack_noctxt SPWRITE on LR architectures On LR architectures, morestack (and morestack_noctxt) are called with a special calling convention, where the caller... — committed to golang/go by cherrymui 2 years ago
[release-branch.go1.19] runtime: mark morestack_noctxt SPWRITE on LR architectures On LR architectures, morestack (and morestack_noctxt) are called with a special calling convention, where the caller... — committed to golang/go by cherrymui 2 years ago
runtime: mark morestack_noctxt SPWRITE on LR architectures On LR architectures, morestack (and morestack_noctxt) are called with a special calling convention, where the caller doesn't save LR on stac... — committed to rajbarik/go by cherrymui 2 years ago
[release-branch.go1.19] runtime: mark morestack_noctxt SPWRITE on LR architectures On LR architectures, morestack (and morestack_noctxt) are called with a special calling convention, where the caller... — committed to tailscale/go by cherrymui 2 years ago
runtime: mark morestack_noctxt SPWRITE for linux/loong64 ref. CL 425396 Updates #54332. Change-Id: I1a235b0cca4dbf79cf61cf5f40b594fc2d940857 Reviewed-on: https://go-review.googlesource.com/c/go/+/4... — committed to golang/go by abner-chenc 2 years ago
runtime: mark morestack_noctxt SPWRITE for linux/loong64 ref. CL 425396 Updates #54332. Change-Id: I1a235b0cca4dbf79cf61cf5f40b594fc2d940857 Reviewed-on: https://go-review.googlesource.com/c/go/+/4... — committed to xen0n/go by abner-chenc 2 years ago

Most upvoted comments

share.getcloudapp.com/p9uQpJww – no crashes today either. the crash on Friday must have been a fluke, but at least we have the telemetry to detect it if it does happen over the next few weeks. I’ll still work on reconstructing the logs from the Friday crash immediately post-deploy though.

Found the problem – the pod was a lingering holdover of the previous build, despite running chronologically long after the new release should have gone. It did not have the patch. You are good to propose the MOVD RSP, RSP patch for 1.20 release and for 1.19 and for 1.18 backport trains.

thank you thank you thank you @cherrymui and @prattmic for your patience.

lizthegrey on Aug 23, 2022

I think I made the hourly cutoff for our build train, will report if I see any crashes this afternoon or over the weekend; if no crashes by Monday, then we can assume that was the fix.

lizthegrey on Aug 19, 2022

diff --git a/src/runtime/asm_arm64.s b/src/runtime/asm_arm64.s
index 7836ba1d96..ec29cd2038 100644
--- a/src/runtime/asm_arm64.s
+++ b/src/runtime/asm_arm64.s
@@ -321,6 +321,7 @@ TEXT runtime·morestack(SB),NOSPLIT|NOFRAME,$0-0
 
 TEXT runtime·morestack_noctxt(SB),NOSPLIT|NOFRAME,$0-0
        MOVW    $0, R26
+       MOVD    RSP, RSP // force SPWITE
        B runtime·morestack(SB)
 
 // spillArgs stores return values from registers to a *internal/abi.RegArgs in R20.

@lizthegrey Could you try if this patch makes any difference? Thanks!

cherrymui on Aug 19, 2022

TEXT runtime.morestack_noctxt.abi0(SB) /usr/local/go/src/runtime/asm_arm64.s
  asm_arm64.s:323       0x7d000                 2a1f03fa                MOVW ZR, R26                    
  asm_arm64.s:324       0x7d004                 17ffffdf                JMP runtime.morestack.abi0(SB)  
  asm_arm64.s:324       0x7d008                 00000000                ?                               
  asm_arm64.s:324       0x7d00c                 00000000                ?

TEXT github.com/klauspost/compress/zstd.(*frameDec).checkCRC(SB) /mnt/ramdisk/go/pkg/mod/github.com/klauspost/compress@v1.15.9/zstd/framedec.go
  framedec.go:297       0x355d40                f9400b90                MOVD 16(R28), R16                                                               
  framedec.go:297       0x355d44                eb3063ff                CMP R16, RSP                                                                    
  framedec.go:297       0x355d48                54000789                BLS 60(PC)                                                                      
  framedec.go:297       0x355d4c                f81c0ffe                MOVD.W R30, -64(RSP)                                                            
  framedec.go:297       0x355d50                f81f83fd                MOVD R29, -8(RSP)                                                               
  framedec.go:297       0x355d54                d10023fd                SUB $8, RSP, R29                                                                
  framedec.go:298       0x355d58                39482002                MOVBU 520(R0), R2                                                               
  framedec.go:298       0x355d5c                36000642                TBZ $0, R2, 50(PC)                                                              
  framedec.go:297       0x355d60                f90027e0                MOVD R0, 72(RSP)                                                                
  framedec.go:303       0x355d64                f940e802                MOVD 464(R0), R2                                                                
  framedec.go:303       0x355d68                f940ec03                MOVD 472(R0), R3                                                                
  framedec.go:303       0x355d6c                f9401442                MOVD 40(R2), R2                                                                 
  framedec.go:303       0x355d70                b27e03e1                ORR $4, ZR, R1                                                                  
  framedec.go:303       0x355d74                aa0303e0                MOVD R3, R0                                                                     
  framedec.go:303       0x355d78                d63f0040                CALL (R2)                                                                       
  framedec.go:304       0x355d7c                b50004a3                CBNZ R3, 37(PC)                                                                 
  framedec.go:309       0x355d80                f94027e2                MOVD 72(RSP), R2                                                                
  framedec.go:309       0x355d84                3940e043                MOVBU 56(R2), R3                                                                
  framedec.go:309       0x355d88                370003a3                TBNZ $0, R3, 29(PC)                                                             
  bytes.go:20           0x355d8c                f90017e1                MOVD R1, 40(RSP)                                                                
  bytes.go:20           0x355d90                f9001be0                MOVD R0, 48(RSP)                                                                
  framedec.go:313       0x355d94                b90027ff                MOVW ZR, 36(RSP)                                                                
  framedec.go:314       0x355d98                f9402040                MOVD 64(R2), R0                                                                 
  framedec.go:314       0x355d9c                97ff8c75                CALL github.com/klauspost/compress/zstd/internal/xxhash.(*Digest).Sum64(SB)     
  framedec.go:319       0x355da0                b90027e0                MOVW R0, 36(RSP)                                                                
  bytes.go:20           0x355da4                f94017e1                MOVD 40(RSP), R1                                                                
  bytes.go:20           0x355da8                f100103f                CMP $4, R1                                                                      
  bytes.go:20           0x355dac                540000c1                BNE 6(PC)                                                                       
  bytes.go:20           0x355db0                910093e0                ADD $36, RSP, R0                                                                
  bytes.go:20           0x355db4                f9401be1                MOVD 48(RSP), R1                                                                
  bytes.go:20           0x355db8                b27e03e2                ORR $4, ZR, R2                                                                  
  bytes.go:20           0x355dbc                97f2f341                CALL runtime.memequal(SB)                                                       
  framedec.go:321       0x355dc0                37000140                TBNZ $0, R0, 10(PC)                                                             
  framedec.go:325       0x355dc4                f0013ffb                ADRP 41938944(PC), R27                                                          
  framedec.go:325       0x355dc8                911fc37b                ADD $2032, R27, R27                                                             
  framedec.go:325       0x355dcc                f9400360                MOVD (R27), R0                                                                  
  framedec.go:325       0x355dd0                f0013ffb                ADRP 41938944(PC), R27                                                          
  framedec.go:325       0x355dd4                911fe37b                ADD $2040, R27, R27                                                             
  framedec.go:325       0x355dd8                f9400361                MOVD (R27), R1                                                                  
  framedec.go:325       0x355ddc                a97ffbfd                LDP -8(RSP), (R29, R30)                                                         
  framedec.go:325       0x355de0                910103ff                ADD $64, RSP, RSP                                                               
  framedec.go:325       0x355de4                d65f03c0                RET                                                                             
  framedec.go:330       0x355de8                d2800000                MOVD $0, R0                                                                     
  framedec.go:330       0x355dec                d2800001                MOVD $0, R1                                                                     
  framedec.go:330       0x355df0                a97ffbfd                LDP -8(RSP), (R29, R30)                                                         
  framedec.go:330       0x355df4                910103ff                ADD $64, RSP, RSP                                                               
  framedec.go:330       0x355df8                d65f03c0                RET                                                                             
  framedec.go:310       0x355dfc                d2800000                MOVD $0, R0                                                                     
  framedec.go:310       0x355e00                d2800001                MOVD $0, R1                                                                     
  framedec.go:310       0x355e04                a97ffbfd                LDP -8(RSP), (R29, R30)                                                         
  framedec.go:310       0x355e08                910103ff                ADD $64, RSP, RSP                                                               
  framedec.go:310       0x355e0c                d65f03c0                RET                                                                             
  framedec.go:306       0x355e10                aa0303e0                MOVD R3, R0                                                                     
  framedec.go:306       0x355e14                aa0403e1                MOVD R4, R1                                                                     
  framedec.go:306       0x355e18                a97ffbfd                LDP -8(RSP), (R29, R30)                                                         
  framedec.go:306       0x355e1c                910103ff                ADD $64, RSP, RSP                                                               
  framedec.go:306       0x355e20                d65f03c0                RET                                                                             
  framedec.go:299       0x355e24                d2800000                MOVD $0, R0                                                                     
  framedec.go:299       0x355e28                d2800001                MOVD $0, R1                                                                     
  framedec.go:299       0x355e2c                a97ffbfd                LDP -8(RSP), (R29, R30)                                                         
  framedec.go:299       0x355e30                910103ff                ADD $64, RSP, RSP                                                               
  framedec.go:299       0x355e34                d65f03c0                RET                                                                             
  framedec.go:297       0x355e38                f90007e0                MOVD R0, 8(RSP)                                                                 
  framedec.go:297       0x355e3c                aa1e03e3                MOVD R30, R3                                                                    
  framedec.go:297       0x355e40                97f49c70                CALL runtime.morestack_noctxt.abi0(SB)                                          
  framedec.go:297       0x355e44                f94007e0                MOVD 8(RSP), R0                                                                 
  framedec.go:297       0x355e48                17ffffbe                JMP github.com/klauspost/compress/zstd.(*frameDec).checkCRC(SB)                 
  framedec.go:297       0x355e4c                00000000                ?

lizthegrey on Aug 19, 2022

It’s possible, though I’m surprised there is no further indication of crash if true. Do you see “stack trace unavailable” at the end? Curiously, I’ve been using the same patch to investigate #54306 and it is working fine.

Oddly, no. What’s also odd is that the kubectl log output stops after a small number of goroutines, rather than printing all the way through. I’ll see if our fluentbit exporter got the entire thing and report back.

On another note, I’ve noticed that most of the stacks so far seem to be in code around github.com/Shopify/sarama.(*RecordBatch).decode (and various other sarama decode functions) which calls into github.com/klauspost/compress/zstd functions. Do you know (from profiles or intuition) if this code path is very hot in this application (so crashing here is likely just coincidence), or fairly rare (so perhaps not a coincidence)? There was one crash above in github.com/honeycombio/libhoney-go, so I lean towards coincidence.

It’s a coincidence, sarama/zstd is about 30-40% of the workload of this app, and libhoney-go is 10%.

lizthegrey on Aug 18, 2022

Thanks. By the way, do you happen to be using musl libc (/Alpine Linux)? (Is this even a cgo binary?)

I’m investigating #54306 in parallel, presumably an issue with musl. It is probably unrelated, but it would help to know if you are using musl.

No, we run on Ubuntu 22.04 containers with standard libc 2.35-0ubuntu3.1. No use of cgo

lizthegrey on Aug 17, 2022

Also repros with go1.19.0

runtime: traceback stuck. pc=0x58e9e0 sp=0x4035cfada0
stack: frame={sp:0x4035cfada0, fp:0x4035cfada0} stack=[0x4035cf8000,0x4035cfc000)
0x0000004035cfaca0:  0x0000004047098bd0  0x000000010001aae8 
0x0000004035cfacb0:  0xc90000000001bfb0  0x0000ffff94c3f878 
0x0000004035cfacc0:  0xc9c4a878b22454ee  0x0000000000000006 
0x0000004035cfacd0:  0x000000000058db7c <github.com/honeycombio/libhoney-go.(*fieldHolder).AddField+0x000000000000004c>  0x000000403cd55520 
0x0000004035cface0:  0x000000000058dbf0 <github.com/honeycombio/libhoney-go.(*fieldHolder).AddField+0x00000000000000c0>  0x0000004035cfad38 
0x0000004035cfacf0:  0x00000040470acb20  0x0000004035cfad58 
0x0000004035cfad00:  0x000000000058e980 <github.com/honeycombio/libhoney-go.(*Event).AddField+0x0000000000000110>  0x0000000000fcae00 
0x0000004035cfad10:  0x0000004047098bd0  0x00000040469ff6c8 
0x0000004035cfad20:  0x0000000000000014  0x0100000000000014 
0x0000004035cfad30:  0x00000040470b2248  0x0000000000ebb3a0 
0x0000004035cfad40:  0x000000000058dc50 <github.com/honeycombio/libhoney-go.(*fieldHolder).AddField.func1+0x0000000000000000>  0x00000040470b2248 
0x0000004035cfad50:  0x0000004035cfad40  0x0000004035cfadb8 
0x0000004035cfad60:  0x00000000005a5898 <github.com/honeycombio/beeline-go/trace.(*Span).AddField+0x0000000000000158>  0x00000040470b2240 
0x0000004035cfad70:  0x00000040469ff6c8  0x0000000000000014 
0x0000004035cfad80:  0x0000000000ebb3a0  0x0000000001ed2ac8 
0x0000004035cfad90:  0x01000000005a5840  0x0000000000f9a880 
0x0000004035cfada0: >0x000000000058e9e0 <github.com/honeycombio/libhoney-go.(*Event).AddField.func1+0x0000000000000000>  0x00000040470b226c 
0x0000004035cfadb0:  0x0000004035cfada0  0x0000004035cfae28 
0x0000004035cfadc0:  0x00000000005a7a4c <github.com/honeycombio/beeline-go.AddField+0x00000000000000cc>  0x0000ffff94c3f878 
0x0000004035cfadd0:  0x00000040469ff6c8  0x0000000001ed2ac8 
0x0000004035cfade0:  0x0000000000ebb3a0  0x0000000001ed2ac8 
0x0000004035cfadf0:  0x0000004035cfae08  0x0000004035cfae28 
0x0000004035cfae00:  0x01000000005a7a00  0x0000000000f9a880 
0x0000004035cfae10:  0x00000000005a5900 <github.com/honeycombio/beeline-go/trace.(*Span).AddField.func1+0x0000000000000000>  0x0000004043dc5688 
0x0000004035cfae20:  0x0000004035cfae10  0x0000004035cfae88 
0x0000004035cfae30:  0x0000000000d79884 <github.com/honeycombio/hound/cmd/shepherd/app.(*App).getSchema+0x00000000000000f4>  0x0000004043dc5600 
0x0000004035cfae40:  0x00000040469ff6c8  0x0000000000000014 
0x0000004035cfae50:  0x0000000000ebb3a0  0x0000000001ed2ac8 
0x0000004035cfae60:  0x000000400015aae0  0x0000000000000014 
0x0000004035cfae70:  0x0000000000ebb3a0  0x0000004043dc5600 
0x0000004035cfae80:  0x00000040469ff6c8  0x0000004035cfaf48 
0x0000004035cfae90:  0x0000000000d755bc <github.com/honeycombio/hound/cmd/shepherd/app.(*App).batchProcess.func1+0x000000000000043c>  0x0000000001491d40 
fatal error: traceback stuck

goroutine 0 [idle]:
runtime.throw({0x11b3553?, 0x1ebf180?})
	/usr/local/go/src/runtime/panic.go:1047 +0x40 fp=0x40009ac610 sp=0x40009ac5e0 pc=0x490f0
runtime.gentraceback(0x1f3c7e0?, 0x0?, 0x40009ac9a8?, 0x403cd55520, 0x0, 0x40009ac9a8, 0x40, 0x0, 0x0?, 0x6)
	/usr/local/go/src/runtime/traceback.go:488 +0xde4 fp=0x40009ac940 sp=0x40009ac610 pc=0x703f4
runtime.sigprof(0x7abe0, 0x40009acc48?, 0x0?, 0x403cd55520, 0x40009a4000)
	/usr/local/go/src/runtime/proc.go:4567 +0xe8 fp=0x40009acbb0 sp=0x40009ac940 pc=0x55618
runtime.sighandler(0x1b?, 0x40009a4000?, 0x40009acc98?, 0x40005e4000?)
	/usr/local/go/src/runtime/signal_unix.go:629 +0x574 fp=0x40009acc60 sp=0x40009acbb0 pc=0x60334
runtime.sigtrampgo(0x1b, 0x40009acda0, 0x40009ace20)
	/usr/local/go/src/runtime/signal_unix.go:479 +0x164 fp=0x40009acce0 sp=0x40009acc60 pc=0x5f9c4
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_linux_arm64.s:471 +0x58 fp=0x40009acda0 sp=0x40009acce0 pc=0x7e228

goroutine 1503875 [running]:

runtime: traceback stuck. pc=0x58e9e0 sp=0x400ffaeda0
stack: frame={sp:0x400ffaeda0, fp:0x400ffaeda0} stack=[0x400ffac000,0x400ffb0000)
0x000000400ffaeca0:  0x0000004046ce2240  0x000000010001f0b8 
0x000000400ffaecb0:  0x7300000001243730  0x0000ffff9d953948 
0x000000400ffaecc0:  0xc8f36a7a4ff431e7  0x0000000000000007 
0x000000400ffaecd0:  0x000000000058db7c <github.com/honeycombio/libhoney-go.(*fieldHolder).AddField+0x000000000000004c>  0x000000400cfa5520 
0x000000400ffaece0:  0x000000000058dc00 <github.com/honeycombio/libhoney-go.(*fieldHolder).AddField+0x00000000000000d0>  0x000000400ffaed38 
0x000000400ffaecf0:  0x0000000000065708 <runtime.rawstringtmp+0x0000000000000048>  0x000000400ffaed58 
0x000000400ffaed00:  0x000000000058e980 <github.com/honeycombio/libhoney-go.(*Event).AddField+0x0000000000000110>  0x0000000000fcae00 
0x000000400ffaed10:  0x0000004046ce2240  0x0000004046110d50 
0x000000400ffaed20:  0x0000000000000016  0x0100000000000016 
0x000000400ffaed30:  0x00000040453eaf68  0x0000000000eccea0 
0x000000400ffaed40:  0x000000000058dc50 <github.com/honeycombio/libhoney-go.(*fieldHolder).AddField.func1+0x0000000000000000>  0x00000040453eaf68 
0x000000400ffaed50:  0x000000400ffaed40  0x000000400ffaedb8 
0x000000400ffaed60:  0x00000000005a5898 <github.com/honeycombio/beeline-go/trace.(*Span).AddField+0x0000000000000158>  0x00000040453eaf60 
0x000000400ffaed70:  0x0000004046110d50  0x0000000000000016 
0x000000400ffaed80:  0x0000000000eccea0  0x000000000147d320 
0x000000400ffaed90:  0x01000000005a5840  0x0000000000f9a880 
0x000000400ffaeda0: >0x000000000058e9e0 <github.com/honeycombio/libhoney-go.(*Event).AddField.func1+0x0000000000000000>  0x00000040453eaf8c 
0x000000400ffaedb0:  0x000000400ffaeda0  0x000000400ffaee28 
0x000000400ffaedc0:  0x00000000005a7a4c <github.com/honeycombio/beeline-go.AddField+0x00000000000000cc>  0x0000ffff9d953948 
0x000000400ffaedd0:  0x0000004046110d50  0x000000000147d320 
0x000000400ffaede0:  0x0000000000eccea0  0x000000000147d320 
0x000000400ffaedf0:  0x000000400ffaee08  0x000000400ffaee28 
0x000000400ffaee00:  0x01000000005a7a00  0x0000000000f9a880 
0x000000400ffaee10:  0x00000000005a5900 <github.com/honeycombio/beeline-go/trace.(*Span).AddField.func1+0x0000000000000000>  0x00000040453c2b88 
0x000000400ffaee20:  0x000000400ffaee10  0x000000400ffaee88 
0x000000400ffaee30:  0x0000000000d79a6c <github.com/honeycombio/hound/cmd/shepherd/app.(*App).getSchema+0x00000000000002dc>  0x00000040453c2b00 
0x000000400ffaee40:  0x0000004046110d50  0x0000000000000016 
0x000000400ffaee50:  0x0000000000eccea0  0x000000000147d320 
0x000000400ffaee60:  0x00000040006af0b0  0x0000000000000016 
0x000000400ffaee70:  0x0000000000eccea0  0x00000040453c2b00 
0x000000400ffaee80:  0x0000004046110d50  0x000000400ffaef48 
0x000000400ffaee90:  0x0000000000d755bc <github.com/honeycombio/hound/cmd/shepherd/app.(*App).batchProcess.func1+0x000000000000043c>  0x0000000001491d40 
fatal error: traceback stuck

goroutine 0 [idle]:
runtime.throw({0x11b3553?, 0x1ebf180?})
	/usr/local/go/src/runtime/panic.go:1047 +0x40 fp=0x400078a610 sp=0x400078a5e0 pc=0x490f0
runtime.gentraceback(0x1f3c7e0?, 0x0?, 0x400078a9a8?, 0x400cfa5520, 0x0, 0x400078a9a8, 0x40, 0x0, 0x0?, 0x6)
	/usr/local/go/src/runtime/traceback.go:488 +0xde4 fp=0x400078a940 sp=0x400078a610 pc=0x703f4
runtime.sigprof(0x7abe4, 0x400078ac48?, 0x0?, 0x400cfa5520, 0x4000780000)
	/usr/local/go/src/runtime/proc.go:4567 +0xe8 fp=0x400078abb0 sp=0x400078a940 pc=0x55618
runtime.sighandler(0x1b?, 0x4000780000?, 0x400078ac98?, 0x40007821a0?)
	/usr/local/go/src/runtime/signal_unix.go:629 +0x574 fp=0x400078ac60 sp=0x400078abb0 pc=0x60334
runtime.sigtrampgo(0x1b, 0x400078ada0, 0x400078ae20)
	/usr/local/go/src/runtime/signal_unix.go:479 +0x164 fp=0x400078ace0 sp=0x400078ac60 pc=0x5f9c4
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_linux_arm64.s:471 +0x58 fp=0x400078ada0 sp=0x400078ace0 pc=0x7e228

goroutine 16981043 [running]:

go tool addr2line shepherd
0x7abe4
0x7abe0
runtime.morestack_noctxt
/usr/local/go/src/runtime/asm_arm64.s:324
runtime.morestack_noctxt
/usr/local/go/src/runtime/asm_arm64.s:323

https://github.com/golang/go/blob/go1.19/src/runtime/asm_arm64.s#L323-L324

lizthegrey on Aug 9, 2022