go: runtime: "fatal: morestack on g0" on Linux

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (`go version`)?

go version go1.9.1 linux/amd64

Does this issue reproduce with the latest release?

I do not know how to reproduce it. But it occurs twice.

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build898823429=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

Run a binary as daemon service.

What did you expect to see?

The process runs forever.

What did you see instead?

I get “fatal: morestack on g0” from stderr. The process is still there but does not respond anymore. When I use curl http://ip:port/debug/pprof/goroutine?debug=1 to check the stack, but it halts. There is nothing useful in stderr or dmesg.

I tried pstack and got the following result. It is very strange that there is only one thread alive. I have googled it but get nothing useful.

Thread 1 (process 12230):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439
#1  0x00000000004293f2 in runtime.futexsleep (addr=0x1b0a950 <runtime.m0+272>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:45
#2  0x000000000040f9ab in runtime.notesleep (n=0x1b0a950 <runtime.m0+272>) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x00000000004315f5 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1680
#4  0x00000000004327d2 in runtime.findrunnable (gp=0x45cbd1 <runtime.goexit+1>, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2135
#5  0x000000000043328c in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2255
#6  0x00000000004335a6 in runtime.park_m (gp=0xca01842780) at /usr/local/go/src/runtime/proc.go:2318
#7  0x0000000000459ffb in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:286
#8  0x0000000001b0a100 in ?? ()
#9  0x00007ffdc5be26e0 in ?? ()
#10 0x0000000001b0a100 in ?? ()
#11 0x00007ffdc5be26d0 in ?? ()
#12 0x0000000000430544 in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1152
#13 0x0000000000459ec1 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:186
#14 0x000000000000000a in ?? ()
#15 0x00007ffdc5be2718 in ?? ()
#16 0x000000000000000a in ?? ()
#17 0x00007ffdc5be2718 in ?? ()
#18 0x0000000000000000 in ?? ()

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 41 (18 by maintainers)

Commits related to this issue

runtime: never allocate during an unrecoverable panic Currently, startpanic_m (which prepares for an unrecoverable panic) goes out of its way to make it possible to allocate during panic handling by ... — committed to golang/go by aclements 6 years ago

Most upvoted comments

@CAFxX I believe that that is https://github.com/golang/go/issues/25229

josharian on May 28, 2018

morestack performs an INT $3 after calling badmorestackg0 to print the “morestack on g0” message. That’s probably where the SIGTRAP is coming from. Though, again, that means the SIGTRAP, thread 2’s stack, and the deadlock attempting to allocmcache from the signal handler are just side-effects of the real problem.

It does mean that whatever went wrong went wrong on thread 2. I’m surprised gdb couldn’t walk over the signal stack switch, but we might be able to get enough from the signal context to unwind the stack ourselves.

@shenli, I assume you’re doing this all from a core file? Or do you still have the process around? Either way, could you try running:

thread 2
frame 7
print/x *info
print/x *ctxt
print/x *gp

aclements on Jan 9, 2018

It’s normal for an inactive goroutine to be sitting in gopark. It’s normal for an inactive thread to be waiting on a futex.

Your original report, morestack on g0, is a clear sign of a bug or of memory corruption. A deadlock could simply be the result of a program bug.

You suggest that a g0 could crash, causing deadlock, but in general that can’t happen. If a g0 crashes, the entire program crashes, as in your original report.

ianlancetaylor on Jan 8, 2018