go: runtime: intermittent os/exec.Command.Start() Hang on Darwin in Presence of "plugin" Package
What version of Go are you using (go version)?
$ go version go version go1.14.2 darwin/amd64 $
Does this issue reproduce with the latest release?
Yes. Also with go version devel +be08e10b3b Fri May 1 21:57:29 2020 +0000 darwin/amd6.
It does not reproduce with Go 1.13.10.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="/Users/craig/go/bin" GOCACHE="/Users/craig/Library/Caches/go-build" GOENV="/Users/craig/Library/Application Support/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GOINSECURE="" GOMODCACHE="/Users/craig/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/craig/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/Users/craig/github/golang/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/Users/craig/github/golang/go/pkg/tool/darwin_amd64" GCCGO="gccgo" GOAMD64="alignedjumps" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/m_/0sxkm11s5wd625ypnp0g0ffm0000gn/T/go-build624939018=/tmp/go-build -gno-record-gcc-switches -fno-common" $
Note that, in a different shell, GOROOT=/usr/local/go, corresponding to Go 1.14.2, where the same problem occurs (though, anecdotally, less frequently) compared to the recent commit, identified above, on master.
What did you do?
Built a simple Go program, ran it repeatedly via:
$ while true; do ./hangme 100; date; done
What did you expect to see?
Repeated runs ad infinitum, no hang.
What did you see instead?
Occasionally (intermittently), the program hangs. SIGQUIT stack dump shows it usually hangs in the Command.Start() receiver in os/exec, which is not supposed to hang at all. (I’ve seen somewhat-different stack traces across different programs and built with different versions of Golang, but they all hang at that call or soon after.)
Such hangs have been observed (by me) only in programs that import plugin (even though they don’t use it at all; only its initialization code should run). Comment-out the _ "plugin" import line in the above program, rebuild (via go build), and rerun the ever-looping command, and it runs until manually stopped.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 5
- Comments: 34 (13 by maintainers)
Commits related to this issue
- [release-branch.go1.16] cmd/link: force eager binding when using plugins on darwin When building/using plugins on darwin, we need to use flat namespace so the same symbol from the main executable and... — committed to golang/go by cherrymui 3 years ago
- [release-branch.go1.17] cmd/link: force eager binding when using plugins on darwin When building/using plugins on darwin, we need to use flat namespace so the same symbol from the main executable and... — committed to golang/go by cherrymui 3 years ago
Awesome, what teamwork! Knowing what’s wrong, and that there’s a fix in a future Go, means we can work around it for the time being.
Background: I maintain a fork/branch of Joker (a Clojure interpreter/clone written in Go) that “autowraps” much of the Go standard library for easy access by Joker code.
A couple of weeks ago, I first noticed that the (small) test suite occasionally hung. Digging further, I posted this to get some feedback:
https://stackoverflow.com/questions/61342000/why-might-exec-command-start-hang-on-darwin
As suggested there, I ran
git bisect runon the Go source code to find “the” commit that introduced or triggered the problem. The culprit simply turned on “new” timer code (turning off old code), which was quite suggestive.Later, I boiled the pertinent Joker code down to a small test program and was able to reproduce that. The first such version pulled in all the Go standard library that my fork of Joker did. I then performed manual bisections of that set and narrowed them down to just the
pluginpackage: when included (despite not being referenced), the sample program would occasionally hang; without it, it wouldn’t.This issue is being tracked (on my end) via https://github.com/jcburley/joker/issues/19. It’s Closed due to a workaround being implemented such that
pluginisn’t included in the list of imported (and wrapped) packages when Joker is built for OS X (Darwin).A slightly “fuller” version of the sample program, along with instructions on another approach to run it, is here:
https://github.com/jcburley/hangme/blob/master/README.md