go: runtime: crash with "invalid pc-encoded table" in TSAN mode
What version of Go are you using (go version)?
go 1.11
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/users/kai.hayashi/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build387775402=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
What did you do?
I am running a fairly run-of-the-mill production api server. I have observed one crash so far on this code since upgrading to go1.11 with fatal error: invalid runtime symbol table. I am running a library in this server code that periodically captures a cpu, block, and mutex profile every few minutes. The sample rates on the mutex profile is 1/10 and 1 block profile every second.
What did you expect to see?
No crashes
What did you see instead?
runtime: invalid pc-encoded table [...]
fatal error: invalid runtime symbol table
goroutine 0 [idle]:
runtime.throw(0x10501de, 0x1c)
/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc00064f258 sp=0xc00064f228 pc=0x42c2b2
runtime.pcvalue(0x179ed90, 0x19f6800, 0xXXXXXXXX005ea0e7, 0xaaf94c, 0xc00064f480, 0xXXXXXXXXXXXXXX01, 0xXXXXXXXX00000000)
/usr/local/go/src/runtime/symtab.go:791 +0x50a fp=0xc00064f300 sp=0xc00064f258 pc=0x449d3a
runtime.funcspdelta(0x179ed90, 0x19f6800, 0xaaf94c, 0xc00064f480, 0xXXXXXXXX00000000)
/usr/local/go/src/runtime/symtab.go:843 +0x5f fp=0xc00064f370 sp=0xc00064f300 pc=0x44a1ef
runtime.gentraceback(0x45d5d5, 0xc0005f3e00, 0x0, 0xc0004d1080, 0x0, 0xc00064f6d8, 0x40, 0x0, 0x0, 0x6, ...)
/usr/local/go/src/runtime/traceback.go:208 +0x1828 fp=0xc00064f678 sp=0xc00064f370 pc=0x450ca8
runtime.sigprof(0x45d5d5, 0xc0005f3e00, 0x0, 0xc0004d1080, 0xc000244380)
/usr/local/go/src/runtime/proc.go:3823 +0x4b6 fp=0xc00064f928 sp=0xc00064f678 pc=0x436cd6
runtime.sighandler(0xc00000001b, 0xc00064fbf0, 0xc00064fac0, 0xc0004d1080)
/usr/local/go/src/runtime/signal_sighandler.go:38 +0x73c fp=0xc00064f9b8 sp=0xc00064f928 pc=0x440dec
This crash is reminiscent of a bug I was hoping would be fixed with the 1.11 release related to https://github.com/golang/go/issues/24925. The crashes do seem less frequent than when I was running the same code under 1.10.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 42 (27 by maintainers)
@scarbo87 I was running into this issue regularly when I had a background profiling process running periodically on a long-running process. My only fix so far has been to shift to only collecting profiles on demand or when prompted by specific anomalous events. Does not really solve the issue but significantly reduces the chance of tripping it…
Indeed. I think this issue has ended up tracking something like three distinct but time-overlapping bugs with similar symptoms. (Hopefully we’ll eventually reach a state where we don’t have so many time-overlapping bugs and they’ll be easier to distinguish! 😅)
There are different failure modes, albeit all being related “invalid pc-encoded table” and profiling signals. The last two seem specific to TSAN, due to its delayed delivery of signals. The problem is that when the signal is delivered the signal context no longer matches the thread’s state, so unwinding would fail.
The earlier ones are probably related to VDSO.
Hi to everyone, I catched the same problem:
I used go 1.12.7 version What should i do?