go: runtime: macOS-only segfault on 1.14+ with "split stack overflow"

What version of Go are you using (`go version`)?

$ go version
go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/alex/Library/Caches/go-build"
GOENV="/Users/alex/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/alex/code/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.2_1/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/56/nb9q27rn4r7_d7w02sv74tjm0000gn/T/go-build443341182=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I created a local main.go like so:

package main

// #include <wasm.h>
//
// static wasm_trap_t* myTrampoline(
//    const wasm_val_t *args,
//    wasm_val_t *results
// ) {
//   return NULL;
// }
//
// void my_run() {
//   wasm_engine_t *engine = wasm_engine_new();
//   wasm_store_t *store = wasm_store_new(engine);
//   wasm_functype_t *ty = wasm_functype_new_0_0();
//   wasm_func_t *func = wasm_func_new(store, ty, myTrampoline);
//   wasm_func_call(func, NULL, NULL);
// }
import "C"
import "runtime"

func main() {
	C.my_run()
	runtime.GC()
}

Next I downloaded the latest wasmtime release and extracted it locally:

$ curl -L https://github.com/bytecodealliance/wasmtime/releases/download/dev/wasmtime-dev-x86_64-macos-c-api.tar.xz | tar xJf - --strip-components=1

Next I compiled the local module:

$ CGO_LDFLAGS="`pwd`/lib/libwasmtime.a" CGO_CFLAGS="-I`pwd`/include" go build -o binary

Finally I ran the binary in an infinite loop:

$ while true; do ./binary || break; done

What did you expect to see?

No segfault. Or more specifically for this to basically run infinitely producing no output.

What did you see instead?

Instead I see sporadic crashes. Some I’ve seen are:

fatal error: runtime: split stack overflow

runtime: newstack sp=0x5791928 stack=[0xc000050000, 0xc000050800]
        morebuf={pc:0x4015580 sp:0xc000050690 lr:0x0}
        sched={pc:0x4053df2 sp:0x5791930 lr:0x0 ctxt:0x0}
runtime: gp=0xc000000180, goid=1, gp->status=0x2
 runtime: split stack overflow: 0x5791928 < 0xc000050000
fatal error: runtime: split stack overflow
runtime stack:
runtime.throw(0x4966544, 0x1d)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/panic.go:1116 +0x72
runtime.newstack()
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/stack.go:1002 +0x83e
runtime.morestack()
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:449 +0x8f
goroutine 1 [running]:
runtime.asmcgocall(0x4056290, 0x5791990)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:640 +0x42 fp=0x5791938 sp=0x5791930 pc=0x4053df2
runtime.libcCall(0x10, 0x5791900, 0x0)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sys_darwin.go:46 +0x6c fp=0x5791968 sp=0x5791938 pc=0x404471c
runtime.sighandler(0x4b37d00, 0x5791ee0, 0x5791f48, 0x4b37d00)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:583 +0x143 fp=0x57919e8 sp=0x5791968 pc=0x403cf03
runtime.sigtrampgo(0x10, 0x5791ee0, 0x5791f48)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:453 +0x1dc fp=0x5791a60 sp=0x57919e8 pc=0x403caac
runtime: unexpected return pc for runtime.sigtramp called from 0x5791ee0
stack: frame={sp:0x5791a60, fp:0x5791a70} stack=[0xc000050000,0xc000050800)
runtime.sigtramp(0x5791f48, 0x5791f48, 0xeaa614b3296df86d, 0x0, 0x4973920, 0x0, 0x5791ab0, 0x7fff6f847613, 0x7ffeefbff5e0, 0x0, …)
/usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sys_darwin_amd64.s:229 +0x33 fp=0x5791a70 sp=0x5791a60 pc=0x4056343

fatal error: unexpected signal during runtime execution

fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffffffffffc0 pc=0x403c363]

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigtramp called from 0x5791ee0
stack: frame={sp:0x57919e0, fp:0x57919f0} stack=[0x5782000,0x5792000)
00000000057918e0: 000000000403d43a <runtime.sighandler+1658> 0000000005791948
00000000057918f0: 000000000000000b 000000c000001800
0000000005791900: 0000000005791958 000000000403cc78 <runtime.adjustSignalStack+328>
0000000005791910: 0000000000000000 0000000005791928
0000000005791920: 0000000005791968 0000000005782000
0000000005791930: 0000000000010000 000000c000000000
0000000005791940: 0000000000000000 0000000005791ee0
0000000005791950: 0000000005791f48 00000000057919d0
0000000005791960: 000000000403ca9e <runtime.sigtrampgo+462> 000000000000000b
0000000005791970: 0000000005791ee0 0000000005791f48
0000000005791980: 000000c000001800 0100000000000000
0000000005791990: 000000c000002000 000000c00000a000
0000000005791: 000000c000002380 000000c000002380
00000000057919b0: 0000000000000000 000000c000001800
00000000057919c0: 0000000005791ee0 0000000005791f48
00000000057919d0: 0000000005791a20 0000000004056343 <runtime.sigtramp+51>
00000000057919e0: <000000000000000b !0000000005791ee0
00000000057919f0: >0000000005791f48 0000000005791f48
0000000005791a00: ec09d23eb3e501c1 0000000004b33ee0
0000000005791a10: 0000000004b63600 0000000000000000
0000000005791a20: 0000000005791aa0 000000000439aaa4
0000000005791a30: 0000000005791590 000000c000001800
0000000005791a40: 0000000005791ee0 0000000005791f48
0000000005791a50: 0000000005791aa0 0000000004056343 <runtime.sigtramp+51>
0000000005791a60: 0000000000000010 0000000005791ee0
0000000005791a70: 0000000005791f48 0000000005791f48
0000000005791a80: 0000000005791ee0 0000000b04b33ee0
0000000005791a90: 0000000005791f48 0000000000000000
0000000005791aa0: 0000000005791ab0 00007fff6f8475fd
0000000005791ab0: 000000c0000527c8 0000000000000000
0000000005791ac0: 0000000000000000 0000000000000000
0000000005791ad0: 0000000000000000 000000060000000e
0000000005791ae0: fffffffffffffff0 0000000000000000 runtime.throw(0x4967d1c, 0x2a) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/panic.go:1116 +0x72 runtime.sigpanic() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:679 +0x46a runtime.(*sigctxt).preparePanic(0x5791948, 0xb, 0xc000001800) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_amd64.go:68 +0x93 runtime.sighandler(0xb, 0x5791ee0, 0x5791f48, 0xc000001800) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:578 +0x67a runtime.sigtrampgo(0xb, 0x5791ee0, 0x5791f48) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:452 +0x1ce runtime: unexpected return pc for runtime.sigtramp called from 0x5791ee0 stack: frame={sp:0x57919e0, fp:0x57919f0} stack=[0x5782000,0x5792000) 00000000057918e0: 000000000403d43a <runtime.sighandler+1658> 0000000005791948 00000000057918f0: 000000000000000b 000000c000001800 0000000005791900: 0000000005791958 000000000403cc78 <runtime.adjustSignalStack+328> 0000000005791910: 0000000000000000 0000000005791928 0000000005791920: 0000000005791968 0000000005782000 0000000005791930: 0000000000010000 000000c000000000 0000000005791940: 0000000000000000 0000000005791ee0 0000000005791950: 0000000005791f48 00000000057919d0 0000000005791960: 000000000403ca9e <runtime.sigtrampgo+462> 000000000000000b 0000000005791970: 0000000005791ee0 0000000005791f48 0000000005791980: 000000c000001800 0100000000000000 0000000005791990: 000000c000002000 000000c00000a000 00000000057919a0: 000000c000002380 000000c000002380 00000000057919b0: 0000000000000000 000000c000001800 00000000057919c0: 0000000005791ee0 0000000005791f48 00000000057919d0: 0000000005791a20 0000000004056343 <runtime.sigtramp+51> 00000000057919e0: <000000000000000b !0000000005791ee0 00000000057919f0: >0000000005791f48 0000000005791f48 0000000005791a00: ec09d23eb3e501c1 0000000004b33ee0 0000000005791a10: 0000000004b63600 0000000000000000 0000000005791a20: 0000000005791aa0 000000000439aaa4 0000000005791a30: 0000000005791590 000000c000001800 0000000005791a40: 0000000005791ee0 0000000005791f48 0000000005791a50: 0000000005791aa0 0000000004056343 <runtime.sigtramp+51> 0000000005791a60: 0000000000000010 0000000005791ee0 0000000005791a70: 0000000005791f48 0000000005791f48 0000000005791a80: 0000000005791ee0 0000000b04b33ee0 0000000005791a90: 0000000005791f48 0000000000000000 0000000005791aa0: 0000000005791ab0 00007fff6f8475fd 0000000005791ab0: 000000c0000527c8 0000000000000000 0000000005791ac0: 0000000000000000 0000000000000000 0000000005791ad0: 0000000000000000 000000060000000e 0000000005791ae0: fffffffffffffff0 0000000000000000 runtime.sigtramp(0x5791f48, 0x5791f48, 0xec09d23eb3e501c1, 0x4b33ee0, 0x4b63600, 0x0, 0x5791aa0, 0x439aaa4, 0x5791590, 0xc000001800, …) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sys_darwin_amd64.s:229 +0x33

goroutine 5 [running]: runtime.asmcgocall(0x4056290, 0x5791910) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:640 +0x42 fp=0x57918b8 sp=0x57918b0 pc=0x4053df2 runtime.(*sigctxt).preparePanic(0x5791948, 0xb, 0xc000001800) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_amd64.go:66 +0x93 fp=0x57918e8 sp=0x57918b8 pc=0x403c363 runtime.sighandler(0xb, 0x5791ee0, 0x5791f48, 0xc000001800) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:578 +0x67a fp=0x5791968 sp=0x57918e8 pc=0x403d43a runtime.sigtrampgo(0xb, 0x5791ee0, 0x5791f48) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:452 +0x1ce fp=0x57919e0 sp=0x5791968 pc=0x403ca9e runtime: unexpected return pc for runtime.sigtramp called from 0x5791ee0 stack: frame={sp:0x57919e0, fp:0x57919f0} stack=[0xc000052000,0xc000052800)

runtime.sigtramp(0x5791f48, 0x5791f48, 0xec09d23eb3e501c1, 0x4b33ee0, 0x4b63600, 0x0, 0x5791aa0, 0x439aaa4, 0x5791590, 0xc000001800, …) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sys_darwin_amd64.s:229 +0x33 fp=0x57919f0 sp=0x57919e0 pc=0x4056343 created by runtime.gcBgMarkStartWorkers /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgc.go:1821 +0x77

goroutine 1 [wait for GC cycle]: runtime.GC() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgc.go:1099 +0x64 main.main() /Users/alex/code/wut/foo.go:24 +0x25

fatal error: runtime: stack split at bad time

runtime: newstack at runtime.goPanicIndex+0xa9 sp=0x5791960 stack=[0xc000050000, 0xc000050800] morebuf={pc:0x403d560 sp:0x5791968 lr:0x0} sched={pc:0x40283b9 sp:0x5791960 lr:0x0 ctxt:0x0} os/signal.signal_ignored(...) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sigqueue.go:265 runtime.sighandler(0xc000000180, 0x5791ee0, 0x5791f48, 0xc000000180) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:588 +0x7a0 fp=0x57919e8 sp=0x5791968 pc=0x403d560 runtime.sigtrampgo(0x10, 0x5791ee0, 0x5791f48) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:453 +0x1dc fp=0x5791a60 sp=0x57919e8 pc=0x403caac runtime: unexpected return pc for runtime.sigtramp called from 0x5791ee0 stack: frame={sp:0x5791a60, fp:0x5791a70} stack=[0xc000050000,0xc000050800)

runtime.sigtramp(0x5791f48, 0x5791f48, 0x7e0fb3c827040376, 0x0, 0x4973920, 0x0, 0x5791ab0, 0x7fff6f847613, 0xc000050640, 0x0, …) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sys_darwin_amd64.s:229 +0x33 fp=0x5791a70 sp=0x5791a60 pc=0x4056343 fatal error: runtime: stack split at bad time

runtime stack: runtime.throw(0x4966d45, 0x20) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/panic.go:1116 +0x72 fp=0x7ffeefbff760 sp=0x7ffeefbff730 pc=0x402a5a2 runtime.newstack() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/stack.go:951 +0xb42 fp=0x7ffeefbff8f0 sp=0x7ffeefbff760 pc=0x4041352 runtime.morestack() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:449 +0x8f fp=0x7ffeefbff8f8 sp=0x7ffeefbff8f0 pc=0x405271f

goroutine 1 [syscall]: os/signal.signal_ignored(…) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/sigqueue.go:265 runtime.sighandler(0xc000000180, 0x5791ee0, 0x5791f48, 0xc000000180) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:588 +0x7a0 fp=0x57919e8 sp=0x5791968 pc=0x403d560 runtime.sigtrampgo(0x10, 0x5791ee0, 0x5791f48) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/signal_unix.go:453 +0x1dc fp=0x5791a60 sp=0x57919e8 pc=0x403caac runtime: unexpected return pc for runtime.sigtramp called from 0x5791ee0 stack: frame={sp:0x5791a60, fp:0x5791a70} stack=[0xc000050000,0xc000050800)

goroutine 2 [force gc (idle)]: runtime.gopark(0x4968c50, 0x4b37870, 0x1411, 0x1) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:304 +0xe0 fp=0xc000050fb0 sp=0xc000050f90 pc=0x402cff0 runtime.goparkunlock(…) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:310 runtime.forcegchelper() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:253 +0xb7 fp=0xc000050fe0 sp=0xc000050fb0 pc=0x402cea7 runtime.goexit() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000050fe8 sp=0xc000050fe0 pc=0x40546b1 created by runtime.init.6 /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]: runtime.gopark(0x4968c50, 0x4b379a0, 0x140c, 0x1) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:304 +0xe0 fp=0xc0000517a8 sp=0xc000051788 pc=0x402cff0 runtime.goparkunlock(…) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:310 runtime.bgsweep(0xc000076000) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgcsweep.go:70 +0x9c fp=0xc0000517d8 sp=0xc0000517a8 pc=0x401c09c runtime.goexit() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc0000517e0 sp=0xc0000517d8 pc=0x40546b1 created by runtime.gcenable /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgc.go:214 +0x5c

goroutine 4 [GC scavenge wait]: runtime.gopark(0x4968c50, 0x4b37960, 0x140d, 0x1) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:304 +0xe0 fp=0xc000051f78 sp=0xc000051f58 pc=0x402cff0 runtime.goparkunlock(…) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:310 runtime.bgscavenge(0xc000076000) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgcscavenge.go:237 +0xd0 fp=0xc000051fd8 sp=0xc000051f78 pc=0x401a690 runtime.goexit() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000051fe0 sp=0xc000051fd8 pc=0x40546b1 created by runtime.gcenable /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgc.go:215 +0x7e

goroutine 18 [GC worker (idle)]: runtime.gopark(0x4968ae8, 0xc0000140b0, 0x1418, 0x0) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/proc.go:304 +0xe0 fp=0xc00004c760 sp=0xc00004c740 pc=0x402cff0 runtime.gcBgMarkWorker(0xc000024000) /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgc.go:1873 +0xff fp=0xc00004c7d8 sp=0xc00004c760 pc=0x40156bf runtime.goexit() /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00004c7e0 sp=0xc00004c7d8 pc=0x40546b1 created by runtime.gcBgMarkStartWorkers /usr/local/Cellar/go/1.14.2_1/libexec/src/runtime/mgc.go:1821 +0x77

This was originally reported upstream in https://github.com/bytecodealliance/wasmtime-go/issues/10, and we’ve been trying to narrow it down. With some investigation we found out that Go 1.13 runs this code successfully. We’ve also got the same code running succesfully on other platforms.

I realize though that this isn’t the best bug report, unfortunately. The native library, wasmtime, is a pretty large project and is a giant wad of compiled Rust code. I’ve tried replacing it with a trivial C implementation to remove the dependency, but then the crash goes away. It seems that the bug here is related to something that the native binary is doing. I’m pretty certain that the fault does not lie in the native binary (e.g. no segfault or out of bounds writes or anything like that), but as with all native code I can’t really entirely rule it out. I’m opening this because at this point we’ve at least narrowed it down to a regression between Go versions, and I’m hoping that folks more knowledgeable with changes could help out?

Is there a way we could help to reduce this further to a bite-sized test case? Or would it be helpful to perhaps bisect the Go release to try to find a revision which caused the segfault to appear here? I’m happy to help out in reducing this further!

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 5
Comments: 35 (19 by maintainers)

Commits related to this issue

Set Go 1.13.10 as the default Go version until macOS segfault is fixed See upstream bytecodealliance/wasmtime-go#10 and golang/go#39079 — committed to avidal/fastlike by avidal 4 years ago
Set Go 1.13.10 as the default Go version until macOS segfault is fixed See upstream bytecodealliance/wasmtime-go#10 and golang/go#39079 — committed to avidal/fastlike by avidal 4 years ago
Set Go 1.13.10 as the default Go version until macOS segfault is fixed See upstream bytecodealliance/wasmtime-go#10 and golang/go#39079 — committed to avidal/fastlike by avidal 4 years ago
[release-branch.go1.14] runtime: set g to gsignal before adjustSignalStack When a signal is received, the runtime probes whether an alternate signal stack is set, if so, adjust gsignal's stack to poi... — committed to golang/go by cherrymui 4 years ago
Update Go versions verified in CI, clarify 1.14 on macOS warning The fix for golang/go#39079 has been backported to 1.14 as of version 1.14.11 so the macOS warning can be clarified and the build matr... — committed to avidal/wasmtime-go by avidal 4 years ago
Update Go versions verified in CI, clarify 1.14 on macOS warning (#40) The fix for golang/go#39079 has been backported to 1.14 as of version 1.14.11 so the macOS warning can be clarified and the bui... — committed to bytecodealliance/wasmtime-go by avidal 4 years ago

Most upvoted comments

@albertvaka Since this was about a segfault, which is fixed, please open a new issue about a deadlock you’re seeing, so it can be tracked separately. In the description please also include a reference to this issue. Thanks.

dmitshur on Dec 18, 2020