go: runtime: `fpTracebackPCs` crash on linux/amd64 in various places
I was running the Sweet benchmarks and I encountered a crash in fpTracebackPCs:
Crash in Go compiler during the go-build benchmark building kubelet.
SIGSEGV: segmentation violation
PC=0x458f97 m=2 sigcode=128
goroutine 0 [idle]:
runtime.fpTracebackPCs(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:944
runtime.traceStackID(0x1e32c9ce48d12?, {0x7f957d0a7018, 0x7f9590097100?, 0x80}, 0xffffffffffff8000?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:916 +0x217 fp=0xc000075cf8 sp=0xc000075cb0 pc=0x458f97
runtime.traceEventLocked(0x42606b?, 0x459960?, 0x0?, 0xc0000485d0, 0x12, 0x0, 0x1, {0x0, 0x0, 0x0})
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:759 +0x285 fp=0xc000075d70 sp=0xc000075cf8 pc=0x458565
runtime.traceEvent(0x0?, 0x1, {0x0, 0x0, 0x0})
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:691 +0xa9 fp=0xc000075dd8 sp=0xc000075d70 pc=0x458269
runtime.traceGoPreempt(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:1534
runtime.gopreempt_m(0x1416360?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:3550 +0x45 fp=0xc000075e18 sp=0xc000075dd8 pc=0x43fb05
runtime.newstack()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/stack.go:1072 +0x3cb fp=0xc000075fc8 sp=0xc000075e18 pc=0x44f92b
runtime.morestack()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:593 +0x8f fp=0xc000075fd0 sp=0xc000075fc8 pc=0x46872f
goroutine 864 [running]:
runtime.traceEvent(0x2c?, 0xffffffffffffffff, {0x0, 0x0, 0x0})
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:668 +0xe8 fp=0xc0044c71a0 sp=0xc0044c7198 pc=0x4582a8
runtime.traceGCMarkAssistDone(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:1496
runtime.gcAssistAlloc(0xc002444340)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcmark.go:516 +0x27d fp=0xc0044c7200 sp=0xc0044c71a0 pc=0x41dc9d
runtime.deductAssistCredit(0x4?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/malloc.go:1220 +0x54 fp=0xc0044c7228 sp=0xc0044c7200 pc=0x40c914
runtime.mallocgc(0x5c0, 0xcf5820, 0x1)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/malloc.go:935 +0xc9 fp=0xc0044c7290 sp=0xc0044c7228 pc=0x40c149
runtime.newarray(0xc0047a6306?, 0xc0047a6460?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/malloc.go:1279 +0x45 fp=0xc0044c72b8 sp=0xc0044c7290 pc=0x40ca65
runtime.makeBucketArray(0xcc1b40?, 0xd0?, 0xa4b0d6?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:364 +0x18d fp=0xc0044c72f8 sp=0xc0044c72b8 pc=0x40d80d
runtime.hashGrow(0xc008a9b428?, 0xc008afb4d0)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:1068 +0x72 fp=0xc0044c7338 sp=0xc0044c72f8 pc=0x40f292
runtime.mapassign(0xcc1b40, 0xc008afb4d0, 0xc0033d3200?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:659 +0xf3 fp=0xc0044c73c0 sp=0xc0044c7338 pc=0x40dff3
cmd/compile/internal/ssa.(*slotCanonicalizer).lookup(0xc008a9b578, {0xc008ae84d0, 0xc0001fa480, 0x0, 0xc008af1bc0, 0x0})
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/debug.go:396 +0x25b fp=0xc0044c7488 sp=0xc0044c73c0 pc=0x7059fb
cmd/compile/internal/ssa.PopulateABIInRegArgOps(0xc004493520)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/debug.go:446 +0x12f fp=0xc0044c76e8 sp=0xc0044c7488 pc=0x705baf
cmd/compile/internal/ssa.BuildFuncDebug(0xcc0520?, 0xc004493520, 0x1100000000?, 0x1b?, 0xc0019d5c20)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/debug.go:578 +0x1cb fp=0xc0044c7898 sp=0xc0044c76e8 pc=0x706c8b
cmd/compile/internal/ssagen.genssa(0xc004493520, 0xc008ac7260)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/ssa.go:7225 +0xf7d fp=0xc0044c7ea8 sp=0xc0044c7898 pc=0xa99d7d
cmd/compile/internal/ssagen.Compile(0xc0033d14a0, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:197 +0x26f fp=0xc0044c7f70 sp=0xc0044c7ea8 pc=0xa5b58f
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc0044c7fb0 sp=0xc0044c7f70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc0044c7fe0 sp=0xc0044c7fb0 pc=0xc46d70
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0044c7fe8 sp=0xc0044c7fe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a
goroutine 1 [semacquire]:
runtime.gopark(0x20?, 0xc0000389b8?, 0x0?, 0xb2?, 0x4053e0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc003e13738 sp=0xc003e13718 pc=0x43900e
runtime.goparkunlock(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.semacquire1(0xc003fa23f8, 0xf8?, 0x1, 0x0, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/sema.go:160 +0x20f fp=0xc003e137a8 sp=0xc003e13738 pc=0x449caf
sync.runtime_Semacquire(0xc00338be40?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/sema.go:62 +0x25 fp=0xc003e137e0 sp=0xc003e137a8 pc=0x466c45
sync.(*WaitGroup).Wait(0xc000b70800?)
../../../../tip+trace+nanotime/bin/goroot/src/sync/waitgroup.go:116 +0x48 fp=0xc003e13808 sp=0xc003e137e0 pc=0x4832c8
cmd/compile/internal/gc.compileFunctions()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:196 +0x21d fp=0xc003e13860 sp=0xc003e13808 pc=0xc467bd
cmd/compile/internal/gc.Main(0xd754f0)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/main.go:369 +0x1c6f fp=0xc003e13ee0 sp=0xc003e13860 pc=0xc4906f
main.main()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/main.go:57 +0xf9 fp=0xc003e13f40 sp=0xc003e13ee0 pc=0xc6ff79
runtime.main()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:255 +0x2b2 fp=0xc003e13fe0 sp=0xc003e13f40 pc=0x438bb2
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc003e13fe8 sp=0xc003e13fe0 pc=0x46a441
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005efa8 sp=0xc00005ef88 pc=0x43900e
runtime.goparkunlock(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.forcegchelper()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:310 +0xb3 fp=0xc00005efe0 sp=0xc00005efa8 pc=0x438e73
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x46a441
created by runtime.init.6 in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:298 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005f778 sp=0xc00005f758 pc=0x43900e
runtime.goparkunlock(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.bgsweep(0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcsweep.go:319 +0xdf fp=0xc00005f7c8 sp=0xc00005f778 pc=0x423fbf
runtime.gcenable.func1()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:178 +0x25 fp=0xc00005f7e0 sp=0xc00005f7c8 pc=0x419745
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x46a441
created by runtime.gcenable in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:178 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00007e000?, 0xe8bcd0?, 0x0?, 0x0?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005ff70 sp=0xc00005ff50 pc=0x43900e
runtime.goparkunlock(...)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:392
runtime.(*scavengerState).park(0x13e90e0)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcscavenge.go:399 +0x49 fp=0xc00005ffa0 sp=0xc00005ff70 pc=0x421f49
runtime.bgscavenge(0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgcscavenge.go:632 +0x59 fp=0xc00005ffc8 sp=0xc00005ffa0 pc=0x4224f9
runtime.gcenable.func2()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:179 +0x25 fp=0xc00005ffe0 sp=0xc00005ffc8 pc=0x4196e5
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x46a441
created by runtime.gcenable in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:179 +0xa5
goroutine 17 [finalizer wait]:
runtime.gopark(0x0?, 0xc00013a4f8?, 0xa0?, 0x21?, 0x1000000010?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00011ee28 sp=0xc00011ee08 pc=0x43900e
runtime.runfinq()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mfinal.go:193 +0x107 fp=0xc00011efe0 sp=0xc00011ee28 pc=0x4187e7
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00011efe8 sp=0xc00011efe0 pc=0x46a441
created by runtime.createfing in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mfinal.go:163 +0x3d
goroutine 18 [trace reader (blocked)]:
runtime.gopark(0xc00005a778?, 0xc00012fce0?, 0x18?, 0x24?, 0xfbb9?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005a738 sp=0xc00005a718 pc=0x43900e
runtime.ReadTrace()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace.go:445 +0x31 fp=0xc00005a7a0 sp=0xc00005a738 pc=0x4579b1
runtime/trace.Start.func1()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace/trace.go:130 +0x45 fp=0xc00005a7e0 sp=0xc00005a7a0 pc=0xc45e65
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x46a441
created by runtime/trace.Start in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/trace/trace.go:128 +0xdf
goroutine 24 [select]:
runtime.gopark(0xc00005afb0?, 0x2?, 0x0?, 0x0?, 0xc00005af6c?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005adc8 sp=0xc00005ada8 pc=0x43900e
runtime.selectgo(0xc00005afb0, 0xc00005af68, 0x30?, 0x0, 0xca4f40?, 0x1)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/select.go:327 +0x725 fp=0xc00005aee0 sp=0xc00005adc8 pc=0x448d05
cmd/compile/internal/gc.compileFunctions.func3()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:154 +0x126 fp=0xc00005afe0 sp=0xc00005aee0 pc=0xc46b26
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:147 +0xe7
goroutine 5 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x3?, 0x13?, 0x82?, 0xbfd8e0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005b750 sp=0xc00005b730 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc00005b7e0 sp=0xc00005b750 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005b7e8 sp=0xc00005b7e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 33 [GC worker (idle)]:
runtime.gopark(0x0?, 0xbfd880?, 0x40?, 0x45?, 0xbfd8e0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005bf50 sp=0xc00005bf30 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc00005bfe0 sp=0xc00005bf50 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005bfe8 sp=0xc00005bfe0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x1e32c84698de9?, 0x3?, 0x60?, 0x59?, 0xbfd8e0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc00005c750 sp=0xc00005c730 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc00005c7e0 sp=0xc00005c750 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x1e32c7ea7231e?, 0x3?, 0xb8?, 0xea?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc000060750 sp=0xc000060730 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc0000607e0 sp=0xc000060750 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x1?, 0x3d?, 0x2c?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc000060f50 sp=0xc000060f30 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc000060fe0 sp=0xc000060f50 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x3?, 0xa7?, 0x84?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc000061750 sp=0xc000061730 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc0000617e0 sp=0xc000061750 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 34 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x3?, 0x41?, 0x91?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc003db0750 sp=0xc003db0730 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc003db07e0 sp=0xc003db0750 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc003db07e8 sp=0xc003db07e0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x142dea0?, 0x1?, 0x60?, 0xc6?, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/proc.go:386 +0xce fp=0xc003db0f50 sp=0xc003db0f30 pc=0x43900e
runtime.gcBgMarkWorker()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1286 +0xe5 fp=0xc003db0fe0 sp=0xc003db0f50 pc=0x41b385
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc003db0fe8 sp=0xc003db0fe0 pc=0x46a441
created by runtime.gcBgMarkStartWorkers in goroutine 1
../../../../tip+trace+nanotime/bin/goroot/src/runtime/mgc.go:1210 +0x1c
goroutine 890 [runnable]:
runtime.mapaccess2_fast64(0xcc2b00, 0xc008c27c08, 0x6e7)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/map_fast64.go:53 +0x185 fp=0xc008c27968 sp=0xc008c27960 pc=0x410b85
cmd/compile/internal/ssa.numberLines.func1(...)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/numberlines.go:114
cmd/compile/internal/ssa.numberLines(0xc00262bd40)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/numberlines.go:136 +0x209 fp=0xc008c27ec8 sp=0xc008c27968 pc=0x743229
cmd/compile/internal/ssa.Compile(0xc00262bd40)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/compile.go:97 +0x95e fp=0xc008c2bb80 sp=0xc008c27ec8 pc=0x6f81be
cmd/compile/internal/ssagen.buildssa(0xc00343f4a0, 0x1)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/ssa.go:567 +0x1fc7 fp=0xc008c2bea8 sp=0xc008c2bb80 pc=0xa63ee7
cmd/compile/internal/ssagen.Compile(0xc00343f4a0, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:187 +0x45 fp=0xc008c2bf70 sp=0xc008c2bea8 pc=0xa5b365
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc008c2bfb0 sp=0xc008c2bf70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc008c2bfe0 sp=0xc008c2bfb0 pc=0xc46d70
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc008c2bfe8 sp=0xc008c2bfe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a
goroutine 901 [runnable]:
cmd/internal/obj/x86.padJumpsCtx.reAssemble(0x20?, 0xc00023f480)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/internal/obj/x86/asm6.go:2025 +0xbf fp=0xc008a9bbe8 sp=0xc008a9bbe0 pc=0x6bfedf
cmd/internal/obj/x86.span6(0xc000186400, 0xc000a18380, 0xc0021cf8b0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/internal/obj/x86/asm6.go:2161 +0x9a5 fp=0xc008a9bd78 sp=0xc008a9bbe8 pc=0x6c08a5
cmd/internal/obj.Flushplist(0xc000186400, 0xc008a9bf40, 0xc0021cf8b0, {0x7ffea33c4841, 0x27})
../../../../tip+trace+nanotime/bin/goroot/src/cmd/internal/obj/plist.go:152 +0x849 fp=0xc008a9bea8 sp=0xc008a9bd78 pc=0x541ca9
cmd/compile/internal/objw.(*Progs).Flush(...)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/objw/prog.go:124
cmd/compile/internal/ssagen.Compile(0xc0034ed340, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:212 +0x4a6 fp=0xc008a9bf70 sp=0xc008a9bea8 pc=0xa5b7c6
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc008a9bfb0 sp=0xc008a9bf70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc008a9bfe0 sp=0xc008a9bfb0 pc=0xc46d70
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc008a9bfe8 sp=0xc008a9bfe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a
goroutine 914 [runnable]:
runtime.mapassign(0xcc1660?, 0xc008c8fb60?, 0xc008c8fe00?)
../../../../tip+trace+nanotime/bin/goroot/src/runtime/map.go:579 +0x525 fp=0xc008ca3c78 sp=0xc008ca3c70 pc=0x40e425
cmd/compile/internal/ssa.deadcode(0xc00478f520)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/deadcode.go:233 +0x1085 fp=0xc008ca3ec8 sp=0xc008ca3c78 pc=0x7012c5
cmd/compile/internal/ssa.Compile(0xc00478f520)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssa/compile.go:97 +0x95e fp=0xc008ca7b80 sp=0xc008ca3ec8 pc=0x6f81be
cmd/compile/internal/ssagen.buildssa(0xc0033dc000, 0x3)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/ssa.go:567 +0x1fc7 fp=0xc008ca7ea8 sp=0xc008ca7b80 pc=0xa63ee7
cmd/compile/internal/ssagen.Compile(0xc0033dc000, 0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:187 +0x45 fp=0xc008ca7f70 sp=0xc008ca7ea8 pc=0xa5b365
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:184 +0x34 fp=0xc008ca7fb0 sp=0xc008ca7f70 pc=0xc46954
cmd/compile/internal/gc.compileFunctions.func3.1()
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:166 +0x30 fp=0xc008ca7fe0 sp=0xc008ca7fb0 pc=0xc46d70
runtime.goexit()
../../../../tip+trace+nanotime/bin/goroot/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc008ca7fe8 sp=0xc008ca7fe0 pc=0x46a441
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 24
../../../../tip+trace+nanotime/bin/goroot/src/cmd/compile/internal/gc/compile.go:165 +0x23a
rax 0xa
rbx 0x7f957d0a7018
rcx 0x80
rdx 0x0
rdi 0x80
rsi 0xfa89481e75c98548
rbp 0xc000075ce8
rsp 0xc000075cb0
r8 0xc002444340
r9 0x7f957d0a7028
r10 0x7e
r11 0x48d3294803e2c148
r12 0x12
r13 0x1335d80
r14 0xc0000069c0
r15 0x3
rip 0x458f97
rflags 0x10286
cs 0x33
fs 0x0
gs 0x0
error: exit status 2
error: exit status 1
(This was run with a modified Go tree, but the only change was replacing cputicks with nanotime and removing the trace tick division in in src/runtime/trace.go.)
I don’t think this is easily reproducible yet so I’ll keep running and try to reproduce.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 24 (18 by maintainers)
Commits related to this issue
- runtime: turn frame pointer unwinding for tracing off by default Frame pointer unwinding during execution tracing sometimes crashes. Until this is diagnosed and fixed, it should be turned off by defa... — committed to golang/go by nsrip-dd a year ago
- runtime: add test for systemstack frame pointer adjustment Add TestSystemstackFramePointerAdjust as a regression test for CL 489015. By turning stackPoisonCopy into a var instead of const and introd... — committed to golang/go by felixge a year ago
- runtime: remove systemstack logic from adjustframe Remove logic for skipping some adjustframe logic for systemstack (aka FuncID_systemstack_switch). This was introduced in 2014 by 9198ed4bd6ec7b7dd37... — committed to golang/go by felixge a year ago
Hmmm. Tell us more.
I was also hacking on this last night (mostly using lots of
dlog()andstackPoisonCopy = true) and came to the same conclusion thatshrinkstackwas involved. E.g. the following passes:I also agree with the conclusion from @mknyszek and @cherrymui that it’s the interaction with
systemstackthat is problematic.AFAICT this is a regression from CL 472195.
One thing that still surprised me was that this issue is only observed during stack shrinking and not stack growth. I think the answer to this is that stack growth can’t happen while we’re on the g0 stack, but stack shrinking (via GC) can happen to a goroutine that switched to g0. Does that sound right?
+1 - this sounds like the best approach to me.
Unless somebody is already working on it (please comment), I’ll try to work on a patch for this.
I can reproduce this on
master(at 969ab34e46) for the TestDeferHeapAndStack test in the runtime package. It crashes pretty consistently, though not always at the same place. Here’s the test crashing within atraceGoPreemptcall duringmorestack:go test -trace /dev/null -run TestDeferHeapAndStack
The test does not always crash at the same place:
Another example
My first thought is to double check the various stack-switching routines (systemstack, mcall, morestack) to see if they’re writing stuff where the frame pointer should be, or not setting a new frame pointer up properly when taking over the stack.
I’ve dug into this a little bit. I don’t have any conclusive answer, but I think what’s happening involves an incorrect value ending up in the
RBPregister at some point during a call togcAssistAlloc. Below are some loosely organized notes from my debugging, hopefully pointing us in a good direction. Note that I don’t always get exactly the same crash, but when things do go bad it seems to consistently be aroundgcAssistAlloc.I built the test binary and ran it under gdb, just doing
TestDeferHeapAndStackwith tracing enabled. I got a crash here:The crash is happening during frame pointer unwinding, at this instruction (where we’re looking for the PC relative to the current frame pointer):
Now, we can see right away that curgp.sched.bp (where we’re going to start unwinding) does not look right:
(This makes me wonder why
debugCheckBP = trueisn’t complaining – it even has a check ofcurgp.sched.bp, which presumably should fail here?)Here’s where morestack was called from:
So morestack is being called from runtime.gcMarkDone. This happens at the beginning of gcMarkDone, before the frame is set up. We know that morestack sets up curgp.sched.sp to point the the caller’s frame, just above where the return address is pushed (i.e. the stack pointer as it would have been immediately prior to the CALL instruction, which pushes the return address).
Since we’re at the begging of runtime.gcMarkDone, we know that the stack pointer should thus be pointing to the return address that was pushed by runtime.gcMarkDone’s caller:
Now, we can look at the disassembly of runtime.gcAssistAlloc to see how big of a frame it uses:
So it uses a 0x50 byte frame. Note that this is after pushing RBP to the stack. Let’s see what was saved there. It should be 0x50 bytes off of curgp.sched.sp, plus an additional 8 bytes because we’re pointing to where the return address within gcAssistAlloc is saved before calling gcMarkDone.
Let’s see what the return address of gcAssistAlloc is, which should be an additional 8 bytes above the saved frame pointer:
This looks right! Now, let’s check the saved frame pointer:
This is promising. The address 0xc0003f3880 is within our stack bounds. Let’s see if there’s a return address saved right above this address, assuming it’s a valid frame pointer:
This checks out too!
So, despite curgp.sched.bp being wrong, it looks like the frame pointers saved on the stack are right. So, why is curgp.sched.bp wrong?
Let’s consider how curgp.sched.bp is set. Here’s where it’s set up by morestack:
So, it’s set to be the current value of RBP when morestack is called. Note that morestack is marked NOFRAME, so it doesn’t create a new frame pointer. It’s jumped to from morestack_noctxt:
So, this means we’re going to use the value of BP as it was immediately prior to gcMarkDone being called within gcAssitAlloc. At some point between when gcAssistAlloc was called, and when we reached the gcMarkDone call, something invalid was put in BP. I think based on the analysis above, gcAssistAlloc would have set up the correct frame pointer. My hunch is that something in between overwrote it incorrectly.
One idea I’m trying out: If my hunch is that RBP is invalid at some point after gcAssistAlloc starts, perhaps let’s stop around where we think it might be invalid and see what its value is.
I set a breakpoint in gcAssistAlloc just before gcMarkDone is called. Then I had gdb display RBP and RSP. I think we should always have RBP >= RSP during normal code (so potentially not for a few instructions during stack switching or something). They should probably also be close-ish together, since in theory RBP just points to the top of the frame and RSP points to the bottom.
A typical case looks like this:
As noted above, these values differ by 0x50, the size of the frame that gcAssitAlloc sets up. So this seems to be right.
Here’s one case where that wasn’t true:
Notice how different the two values are.
Here’s the stack trace at this point:
This is where I’m stumped for now. I don’t yet have a machine that can run
rr, but I think that would be helpful to get to a point like this where the frame pointer seems to be invalid and go backwards to the places where it is updated.