go: runtime: unexpected return pc for runtime.sigpanic
This is probably a duplicate of some other recent runtime issues, but it happens to be very easily reproduced using Go 1.10rc1. I’m sure it can be minimized further, but I am almost out of dev time for several days, so I wanted to get it up here ASAP in case it was helpful for @aclements.
Repro: Grab https://gist.github.com/josharian/cec3dbe91b2cffaf7419f53757cce3d3 and run it from inside any git repo. (See https://github.com/libgit2/git2go if there are questions about how to install git2go.) Pasted code:
package main
import (
"os"
git "gopkg.in/libgit2/git2go.v26"
)
func main() {
pwd, err := os.Getwd()
check(err)
repo, err := git.OpenRepositoryExtended(pwd, 0, "")
check(err)
rw, err := repo.Walk()
check(err)
rw.Push(nil)
}
func check(err error) {
if err != nil {
panic(err)
}
}
The resulting crash is obviously programmer error (rw.Push expects a non-nil parameter), but the “unexpected return pc” part shouldn’t be present.
Result:
$ go run x.go
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x42cafcd]
runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x42cafcd
stack: frame={sp:0x7ffeefbff510, fp:0x7ffeefbff560} stack=[0x7ffeefb80350,0x7ffeefbff7d0)
00007ffeefbff410: 00007ffeefbff480 000000000402aeec <runtime.dopanic_m+540>
00007ffeefbff420: 000000000402a991 <runtime.throw+129> 00007ffeefbff4f0
00007ffeefbff430: 0000000000000000 0000000004214f80
00007ffeefbff440: 010000000402b448 0000000000000004
00007ffeefbff450: 000000000000001f 00000000042cafcd
00007ffeefbff460: 0000000000000000 0000000000000001
00007ffeefbff470: 000000000415e881 000000c420000180
00007ffeefbff480: 00007ffeefbff4a8 000000000404f7bc <runtime.dopanic.func1+60>
00007ffeefbff490: 0000000004214f80 000000000402a991 <runtime.throw+129>
00007ffeefbff4a0: 00007ffeefbff4f0 00007ffeefbff4e0
00007ffeefbff4b0: 000000000402a8ba <runtime.dopanic+74> 00007ffeefbff4c0
00007ffeefbff4c0: 000000000404f780 <runtime.dopanic.func1+0> 0000000004214f80
00007ffeefbff4d0: 000000000402a991 <runtime.throw+129> 00007ffeefbff4f0
00007ffeefbff4e0: 00007ffeefbff500 000000000402a991 <runtime.throw+129>
00007ffeefbff4f0: 0000000000000000 000000000000002a
00007ffeefbff500: 00007ffeefbff550 000000000403d39e <runtime.sigpanic+654>
00007ffeefbff510: <0000000004160a85 000000000000002a
00007ffeefbff520: 0000000000000000 0000000000000000
00007ffeefbff530: 0000000000000000 0000000004214f80
00007ffeefbff540: 0000000000000000 0000000000000000
00007ffeefbff550: 00007ffeefbff560 !00000000042cafcd
00007ffeefbff560: >00007ffeefbff600 00000000042c7287
00007ffeefbff570: 0000000000000000 0000000000000000
00007ffeefbff580: 0000000000000000 0000000000000000
00007ffeefbff590: 0000000000000000 0000000000000000
00007ffeefbff5a0: 0000000000000000 0000000004611750
00007ffeefbff5b0: 0000000000000000 0000020100000280
00007ffeefbff5c0: 0000000000000000 0000000004611758
00007ffeefbff5d0: 909597a7fa2e00c2 00000000046369a0
00007ffeefbff5e0: 0000000004611730 0000000000000000
00007ffeefbff5f0: 0000000000000000 00007ffeefbff658
00007ffeefbff600: 00007ffeefbff630 00000000042c71fb
00007ffeefbff610: 0000000000000000 0000000000000028
00007ffeefbff620: 0000000000000000 00007ffeefbff6d8
00007ffeefbff630: 00007ffeefbff6b0 00000000042c568e
00007ffeefbff640: 00000000ffffffff 00000000046369a0
00007ffeefbff650: fffffffeefbff6a0 0000000000000000
runtime.throw(0x4160a85, 0x2a)
/Users/josh/go/1.10/src/runtime/panic.go:619 +0x81
runtime: unexpected return pc for runtime.sigpanic called from 0x42cafcd
stack: frame={sp:0x7ffeefbff510, fp:0x7ffeefbff560} stack=[0x7ffeefb80350,0x7ffeefbff7d0)
00007ffeefbff410: 00007ffeefbff480 000000000402aeec <runtime.dopanic_m+540>
00007ffeefbff420: 000000000402a991 <runtime.throw+129> 00007ffeefbff4f0
00007ffeefbff430: 0000000000000000 0000000004214f80
00007ffeefbff440: 010000000402b448 0000000000000004
00007ffeefbff450: 000000000000001f 00000000042cafcd
00007ffeefbff460: 0000000000000000 0000000000000001
00007ffeefbff470: 000000000415e881 000000c420000180
00007ffeefbff480: 00007ffeefbff4a8 000000000404f7bc <runtime.dopanic.func1+60>
00007ffeefbff490: 0000000004214f80 000000000402a991 <runtime.throw+129>
00007ffeefbff4a0: 00007ffeefbff4f0 00007ffeefbff4e0
00007ffeefbff4b0: 000000000402a8ba <runtime.dopanic+74> 00007ffeefbff4c0
00007ffeefbff4c0: 000000000404f780 <runtime.dopanic.func1+0> 0000000004214f80
00007ffeefbff4d0: 000000000402a991 <runtime.throw+129> 00007ffeefbff4f0
00007ffeefbff4e0: 00007ffeefbff500 000000000402a991 <runtime.throw+129>
00007ffeefbff4f0: 0000000000000000 000000000000002a
00007ffeefbff500: 00007ffeefbff550 000000000403d39e <runtime.sigpanic+654>
00007ffeefbff510: <0000000004160a85 000000000000002a
00007ffeefbff520: 0000000000000000 0000000000000000
00007ffeefbff530: 0000000000000000 0000000004214f80
00007ffeefbff540: 0000000000000000 0000000000000000
00007ffeefbff550: 00007ffeefbff560 !00000000042cafcd
00007ffeefbff560: >00007ffeefbff600 00000000042c7287
00007ffeefbff570: 0000000000000000 0000000000000000
00007ffeefbff580: 0000000000000000 0000000000000000
00007ffeefbff590: 0000000000000000 0000000000000000
00007ffeefbff5a0: 0000000000000000 0000000004611750
00007ffeefbff5b0: 0000000000000000 0000020100000280
00007ffeefbff5c0: 0000000000000000 0000000004611758
00007ffeefbff5d0: 909597a7fa2e00c2 00000000046369a0
00007ffeefbff5e0: 0000000004611730 0000000000000000
00007ffeefbff5f0: 0000000000000000 00007ffeefbff658
00007ffeefbff600: 00007ffeefbff630 00000000042c71fb
00007ffeefbff610: 0000000000000000 0000000000000028
00007ffeefbff620: 0000000000000000 00007ffeefbff6d8
00007ffeefbff630: 00007ffeefbff6b0 00000000042c568e
00007ffeefbff640: 00000000ffffffff 00000000046369a0
00007ffeefbff650: fffffffeefbff6a0 0000000000000000
runtime.sigpanic()
/Users/josh/go/1.10/src/runtime/signal_unix.go:372 +0x28e
goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x410adb0, 0xc420067eb8, 0x4162d70)
/Users/josh/go/1.10/src/runtime/cgocall.go:128 +0x64 fp=0xc420067e88 sp=0xc420067e50 pc=0x4004b64
gopkg.in/libgit2/git2go%2ev26._Cfunc_git_revwalk_push(0x4636760, 0x0, 0x0)
_cgo_gotypes.go:6303 +0x4d fp=0xc420067eb8 sp=0xc420067e88 pc=0x40fcd3d
gopkg.in/libgit2/git2go%2ev26.(*RevWalk).Push.func1(0x4636760, 0x0, 0xc42000e030)
/Users/josh/src/gopkg.in/libgit2/git2go.v26/walk.go:44 +0x60 fp=0xc420067ef0 sp=0xc420067eb8 pc=0x4102dd0
gopkg.in/libgit2/git2go%2ev26.(*RevWalk).Push(0xc4200105a0, 0x0, 0x0, 0x0)
/Users/josh/src/gopkg.in/libgit2/git2go.v26/walk.go:44 +0x65 fp=0xc420067f18 sp=0xc420067ef0 pc=0x4101ee5
main.main()
/Users/josh/go/tip/src/x.go:19 +0xe3 fp=0xc420067f88 sp=0xc420067f18 pc=0x4104023
runtime.main()
/Users/josh/go/1.10/src/runtime/proc.go:198 +0x212 fp=0xc420067fe0 sp=0xc420067f88 pc=0x402c202
runtime.goexit()
/Users/josh/go/1.10/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420067fe8 sp=0xc420067fe0 pc=0x4053491
exit status 2
System details
go version go1.10rc1 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/josh/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/josh"
GORACE=""
GOROOT="/Users/josh/go/1.10"
GOTMPDIR=""
GOTOOLDIR="/Users/josh/go/1.10/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/1t/n61cbvls5bl293bbb0zyypqw0000gn/T/go-build301881037=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.10rc1 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10rc1
uname -v: Darwin Kernel Version 17.3.0: Thu Nov 9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64
ProductName: Mac OS X
ProductVersion: 10.13.2
BuildVersion: 17C205
lldb --version: lldb-900.0.64
Swift-4.0
gdb --version: GNU gdb (GDB) 8.0.1
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 15 (13 by maintainers)
For the original reproducer, the problem is indeed that we’re injecting a
sigpanic
call directly into C, sogentraceback
is seeing a return PC fromsigpanic
that’s in C code. For that, I think Ian’s right that we should just suppress the hex dump if there might be C code on the stack. I’m not sure what else we could do. Maybe we should only do that if we just saw asigpanic
frame since I think that’s the only time this should happen.For the second reproducer, after switching to the system stack,
asmcgocall
aligns the stack before calling C code, sogentraceback
is reading an uninitialized stack slot and treating it as a return PC. That’s not so great and I think we should addasmcgocall
to the set of “top of stack” functions since it does the system stack switch.The difference between the reproducers is rather amusing. In general we should see things like in the original reproducer. However, two GCC optimizations combine to make it appear like
asmcgocall
calledsigpanic
in the traceback of the second reproducer: becauseprintIt
returns void, GCC optimizes the the generated cgo wrapper into a tail call toprintIt
, so the wrapper doesn’t appear on the stack; and becauseprintIt
has a zero-sized frame,printIt
’s own return PC (which isasmcgocall
) appears in the position ofsigpanic
’s return PC, so we also loseprintIt
from the stack. At that point, all of the actual C code is gone from the stack, so we don’t wind up trying to unwind into it. Even changingprintIt
’s return type toint
is enough to inhibit the tail-call optimization and make the second reproducer fail like the first one.It turns out this doesn’t work. If the C code calls back in to Go, we switch back to the regular G stack and if we try to unwind from there, we’ll see the call to
asmcgocall
from before it did the system stack switch. We want to be able to unwind through that to the rest of the regular Go stack, so we can’t just treatasmcgocall
as a top-of-stack function all the time.Interesting. Thanks for the repro. This happened recently on linux/arm on the build dashboard, too: https://build.golang.org/log/4552a24f91dc893759cce93184c415f3986fcbc6
I haven’t dug into this very deeply, but it sort of looks like we’re injecting a
sigpanic
call into C code and then attempting to do a stack walk, which of course fails when it tries to unwind beyond thesigpanic
. I thought we were supposed to just forward signals that came in during C execution, but perhaps that broke./cc @ianlancetaylor @bcmills