go: plugin: net/http.Get fatals with "runtime: unexpected return pc for runtime.goexit called"

What version of Go are you using (go version)?

go version devel +94d7c884c3 Thu Dec 14 14:57:01 2017 +0000 darwin/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/qipeng/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/qipeng/workspace/go"
GORACE=""
GOROOT="/Users/qipeng/program/go"
GOTMPDIR=""
GOTOOLDIR="/Users/qipeng/program/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bs/9120vpt14jd7rkhg27kdbj480000gn/T/go-build502747584=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Call http.Get in plugin. Like: plugin.go

func Init() {
	res, err := http.Get("https://www.google.com")
	if err != nil {
		fmt.Println("Init Error:", err)
	}
	res.Body.Close()
}

main.go

	p, err := plugin.Open("post.so")
	if err != nil {
		panic(err)
	}
	add, err := p.Lookup("Init")
	if err != nil {
		panic(err)
	}
	add.(func())()

DEMO

What did you expect to see?

Get Google Website

What did you see instead?

command-line-arguments
runtime: unexpected return pc for runtime.goexit called from 0x5466c00
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x52e35a4, 0x11)
        /Users/qipeng/program/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42008c900, 0x0, 0x0, 0x7fffffff, 0x52f3658, 0x7fff5fbff078, 0x0, ...)
        /Users/qipeng/program/go/src/runtime/traceback.go:286 +0x1a72
runtime.copystack(0xc42008c900, 0x1000, 0x7fff5fbf0001)
        /Users/qipeng/program/go/src/runtime/stack.go:891 +0x26e
runtime.newstack()
        /Users/qipeng/program/go/src/runtime/stack.go:1056 +0x304
runtime.morestack()
        /Users/qipeng/program/go/src/runtime/asm_amd64.s:480 +0x89

goroutine 19 [copystack]:
runtime.newobject(0x52b0280, 0x0)
        /Users/qipeng/program/go/src/runtime/malloc.go:838 +0x51 fp=0xc420052af8 sp=0xc420052af0 pc=0x5012571
net/http.(*Transport).dialConn(0x5466c00, 0x531aec0, 0xc4200aa030, 0xc42014e080, 0x52e4bf8, 0x5, 0xc4200ec1c0, 0x12, 0x0, 0x0, ...)
        /Users/qipeng/program/go/src/net/http/transport.go:1082 +0x55 fp=0xc420052f38 sp=0xc420052af8 pc=0x522a205
net/http.(*Transport).getConn.func4(0x5466c00, 0x531aec0, 0xc4200aa030, 0xc4200a6f00, 0xc420098180)
        /Users/qipeng/program/go/src/net/http/transport.go:943 +0x76 fp=0xc420052fb8 sp=0xc420052f38 pc=0x5233c56
runtime.goexit()
        /Users/qipeng/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420052fc0 sp=0xc420052fb8 pc=0x5051951
created by net/http.(*Transport).getConn
        /Users/qipeng/program/go/src/net/http/transport.go:942 +0x361

goroutine 1 [select]:
net/http.(*Transport).getConn(0x5466c00, 0xc4200a6ed0, 0xc42014e080, 0x52e4bf8, 0x5, 0xc4200ec1c0, 0x12, 0x0, 0x0, 0xc4200aa3f0)
        /Users/qipeng/program/go/src/net/http/transport.go:948 +0x552
net/http.(*Transport).RoundTrip(0x5466c00, 0xc420150000, 0x5466c00, 0x0, 0x0)
        /Users/qipeng/program/go/src/net/http/transport.go:400 +0x607
net/http.send(0xc420150000, 0x53154a0, 0x5466c00, 0x0, 0x0, 0x0, 0xc4200aeed8, 0xf8, 0xc420067c70, 0x1)
        /Users/qipeng/program/go/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0x546b260, 0xc420150000, 0x0, 0x0, 0x0, 0xc4200aeed8, 0x0, 0x1, 0x5012558)
        /Users/qipeng/program/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0x546b260, 0xc420150000, 0x52e4bf8, 0x16, 0x0)
        /Users/qipeng/program/go/src/net/http/client.go:615 +0x298
net/http.(*Client).Get(0x546b260, 0x52e4bf8, 0x16, 0xc4200a6090, 0xc4200ca240, 0xc4200ca240)
        /Users/qipeng/program/go/src/net/http/client.go:396 +0x9d
net/http.Get(0x52e4bf8, 0x16, 0x1, 0xffffffffffffffff, 0xc420067ef8)
        /Users/qipeng/program/go/src/net/http/client.go:370 +0x44
plugin/unnamed-b216a47df41c38c43adcbf9feb7d3876f589ad7d.Init()
        /Users/qipeng/workspace/go/src/github.com/nzlov/testplugin/post.go:9 +0x3a
main.main()
        /Users/qipeng/workspace/go/src/github.com/nzlov/testplugin/main.go:17 +0xb7

goroutine 18 [finalizer wait]:
runtime.gopark(0x52f3860, 0x40d4f48, 0x52e274a, 0xe, 0x14, 0x1)
        /Users/qipeng/program/go/src/runtime/proc.go:291 +0x126
runtime.goparkunlock(0x40d4f48, 0x52e274a, 0xe, 0x14, 0x1)
        /Users/qipeng/program/go/src/runtime/proc.go:297 +0x5e
runtime.runfinq()
        /Users/qipeng/program/go/src/runtime/mfinal.go:175 +0xbe
runtime.goexit()
        /Users/qipeng/program/go/src/runtime/asm_amd64.s:2361 +0x1
exit status 2

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 4
  • Comments: 32 (22 by maintainers)

Commits related to this issue

Most upvoted comments

If my guess is correct then the simple fix is to introduce a package scope variable initialized to funcPC(goexit) and use that instead of calling funcPC(goexit) each time.

That was the fix I was thinking of, but I was worried that there might be a larger issue that this was just the tip of the iceberg for. But maybe you’re right, the only issue is function address equality, and as long as we don’t rely on that across packages, it’s ok if the plugin and main program are using two different runtimes with identical contents. As long as all the mutable global variables are shared.

goexitPC is already that package scope variable you describe. We just have to use that instead of funcPC(goexit) everywhere. And audit all other uses of funcPC.

This issue still happens in 1.10. and Go 1.10 release notes says that it supports plugin build on Darwin.

We are both in the same (sinking) boat.

I’m not familiar with how dynamic linking works on MacOS. But this looks like the failing code is running inside the plugin. It’s essential for correctness that the main executable and a shared library agree on the location of a global variable. So it’s not surprising that the plugin gets the value of goexitPC correct. It’s much less essential for the main executable and a shared library to agree on the address of a function. For a function it’s important that all calls go to the same place, and in shared libraries that is normally done by having the shared library use a Procedure Linkage Table to branch to the function in the main executable if it is defined there, and to use the copy in the shared library if it is not. But that is only calls, not addresses. ELF dynamic linkers go to considerable effort to ensure that equality comparisons of functions in executables and shared libraries resolve as is required by C. It’s possible that the Darwin dynamic linker does not. In that case, the reference to the function in the plugin may be retrieving the address of the PLT entry, not the address of the function in the main executable. In particular funcPC is going to pull apart the goexit.f value to fetch the PC.

If my guess is correct then the simple fix is to introduce a package scope variable initialized to funcPC(goexit) and use that instead of calling funcPC(goexit) each time. The main executable will set the value to the address in the main executable, and the reference in the shared library will reliably retrieve that value from the main executable. Since the shared library PLT stub should always branch to the main executable version, the global variable should match what is on the stack.

I’ve retitled the bug a little and @juhwany, it’s cool to see you are already here as I just closed your duplicate issue in favor of this already existent one.

I am seeing this as well in a similar situation.

Version: go version go1.10beta1 darwin/amd64

My plugin uses https://github.com/atlassian/go-sentry-api which also appears to use net/http. Would be happy to provide more info if the original post doesn’t provide enough.