go: plugin: SIGSEGV or unknown caller pc on ppc64le with -buildmode pie

What version of Go are you using (`go version`)?

go version go1.15.6 linux/ppc64le

(also seen on now unsupported go version go1.13.12 linux/ppc64le)

Does this issue reproduce with the latest release?

Yes, using 1.15.6 as above.

What operating system and processor architecture are you using (`go env`)?

go env Output

GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/home/dtrudg/.cache/go-build"
GOENV="/home/dtrudg/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/dtrudg/go/pkg/mod"
GONOPROXY="github.com/sylabs"
GONOSUMDB="github.com/sylabs"
GOOS="linux"
GOPATH="/home/dtrudg/go"
GOPRIVATE="github.com/sylabs"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build790835145=/tmp/go-build -gno-record-gcc-switches"

Distro Information

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)
$ uname -a
Linux sylabs01 3.10.0-1062.4.1.el7.ppc64le #1 SMP Wed Sep 25 13:57:38 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
$ lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    8
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Model:                 2.2 (pvr 004e 0202)
Model name:            POWER8 (architected), altivec supported
Hypervisor vendor:     pHyp
Virtualization type:   para
L1d cache:             32K
L1i cache:             32K
NUMA node0 CPU(s):     0-7

What did you do?

Compiled and used a go plugin, where the main executable is compiled using -buildmode=pie on ppc64le.

Plugin code is, unfortunately, closed source. Main executable is built from https://github.com/hpcng/singularity/releases/tag/v3.7.0 - I will try to get a simpler / open-source reproducer when time permits.

What did you expect to see?

No errors. The same configuration works without issue on amd64 and arm64.

On pp64le using -buildmode=default there is no error. Only -buildmode=pie causes issues.

Also, we see no issues with -buildmode=pie on ppc64le when we are not using plugins.

What did you see instead?

SIGSEGV or unknown caller pc are seen in consecutive runs, with failures occuring at slightly different points in execution (of code within the plugin). The traces/errors seem similar, superficially, to those in https://github.com/golang/go/issues/30283 (from https://github.com/containerd/containerd/issues/3005). Those issues occurred on ppc64le with -buildmode=pie (but not plugins afaik) and were fixed in go 1.12.

3 backtraces from consecutive runs:

Backtrace 1

unexpected fault address 0x3fff6cd6da10
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x3fff6cd6da10 pc=0x1297dbdd8]
goroutine 23 [running]:
runtime.throw(0x12a1f6081, 0x5)
runtime/panic.go:1116 +0x68 fp=0xc000449710 sp=0xc0004496d0 pc=0x1297d14b8
runtime.sigpanic()
runtime/signal_unix.go:749 +0x3f8 fp=0xc000449750 sp=0xc000449710 pc=0x1297e9428
runtime.goexit1()
runtime/proc.go:2946 +0x38 fp=0xc0004497b8 sp=0xc000449770 pc=0x1297dbdd8
runtime.goexit()
runtime/asm_ppc64x.s:885 +0x10 fp=0xc0004497b8 sp=0xc0004497b8 pc=0x129809440
created by text/template/parse.lex
text/template/parse/lex.go:223 +0x1b4
goroutine 1 [runnable]:
reflect.funcLayout(0x12a46e0e0, 0x0, 0x1, 0xc00023e160, 0xe, 0x1, 0x194)
reflect/type.go:3052 +0x464
reflect.Value.call(0x12a46e0e0, 0x12a5e2fe8, 0x13, 0x3fff6cacee2e, 0x4, 0xc0000991a0, 0x2, 0x2, 0x2, 0x2, …)
reflect/value.go:434 +0x528
reflect.Value.Call(0x12a46e0e0, 0x12a5e2fe8, 0x13, 0xc00023e160, 0xe, 0xe, 0x12abc7f00, 0x0, 0x0)
reflect/value.go:337 +0xb4
text/template.safeCall(0x12a46e0e0, 0x12a5e2fe8, 0x13, 0xc00023e160, 0xe, 0xe, 0x0, 0x0, 0x0, 0x0, …)
text/template/funcs.go:365 +0xb4
text/template.(*state).evalCall(0xc00035f618, 0x3fff6cb945a0, 0xc000119040, 0x99, 0x12a46e0e0, 0x12a5e2fe8, 0x13, 0x12a62d460, 0xc000098d20, 0xc0004f80f3, …)
text/template/exec.go:720 +0x5c8
text/template.(*state).evalFunction(0xc00035f618, 0x3fff6cb945a0, 0xc000119040, 0x99, 0xc000098de0, 0x12a62d460, 0xc000098d20, 0xc000226700, 0xf, 0x10, …)
text/template/exec.go:575 +0x130
text/template.(*state).evalCommand(0xc00035f618, 0x3fff6cb945a0, 0xc000119040, 0x99, 0xc000098d20, 0x12a498f00, 0x12ac8bfc8, 0x99, 0x0, 0x3fff6c8927f4, …)
text/template/exec.go:462 +0x100
text/template.(*state).evalPipeline(0xc00035f618, 0x3fff6cb945a0, 0xc000119040, 0x99, 0xc0007842a0, 0x3fff6ccd2d00, 0xc000099110, 0x30)
text/template/exec.go:431 +0x360
text/template.(*state).walk(0xc00035f618, 0x3fff6cb945a0, 0xc000119040, 0x99, 0x12a62d2e0, 0xc0000990e0)
text/template/exec.go:255 +0x31c
text/template.(*state).walk(0xc00035f618, 0x3fff6cb945a0, 0xc000119040, 0x99, 0x12a62d640, 0xc000098cc0)
text/template/exec.go:263 +0x12c
text/template.(*Template).execute(0xc0000a6200, 0x3fff6cba84a0, 0xc000784300, 0x3fff6cb945a0, 0xc000119040, 0x0, 0x0)
text/template/exec.go:220 +0x198
text/template.(*Template).Execute(…)
text/template/exec.go:203
github.com/sylabs/singularity/pro-log-plugin.SyslogOutput.Log(0xc00078a1e0, 0xd, 0xc00078a200, 0x6, 0xc00078a220, 0xb, 0xc00078e040, 0x3b, 0xc0000a6200, 0x3ec, …)
github.com/sylabs/singularity/pro-log-plugin/syslog.go:61 +0x17c
github.com/sylabs/singularity/pro-log-plugin.(*pluginImplementation).callOutputs(0xc0000ec120)
github.com/sylabs/singularity/pro-log-plugin/main.go:79 +0x3f8
github.com/sylabs/singularity/pro-log-plugin.callbackLog.func1(0x12abedd80, 0x12ac8bfc8, 0x0, 0x0, 0x0, 0x0)
github.com/sylabs/singularity/pro-log-plugin/main.go:60 +0x164
github.com/spf13/cobra.(*Command).execute(0x12abedd80, 0xc00000e070, 0x0, 0x0, 0x12abedd80, 0xc00000e070)
github.com/spf13/cobra@v1.1.1/command.go:829 +0x4ac
github.com/spf13/cobra.(*Command).ExecuteC(0x12abedae0, 0x12a5e41f8, 0xc000096240, 0xc0002a1620)
github.com/spf13/cobra@v1.1.1/command.go:958 +0x2b4
github.com/spf13/cobra.(*Command).Execute(…)
github.com/spf13/cobra@v1.1.1/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(…)
github.com/spf13/cobra@v1.1.1/command.go:888
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity()
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:431 +0x1b0
main.main()
github.com/sylabs/singularity@v0.0.0/cmd/singularity/cli.go:19 +0x64
goroutine 21 [syscall]:
os/signal.signal_recv(0x0)
runtime/sigqueue.go:147 +0x10c
os/signal.loop()
os/signal/signal_unix.go:23 +0x38
created by os/signal.Notify.func1.1
os/signal/signal.go:150 +0x60
goroutine 22 [select]:
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity.func2(0xc000096240, 0xc0002a1620, 0x12a625ae0, 0xc000407cc0)
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:423 +0xc0
created by github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:422 +0x178

Backtrace 2

runtime: unexpected return pc for runtime.sigpanic called from 0x3fff6f7ee5b8
stack: frame={sp:0x3fff6f7ee5b8, fp:0x3fff6f7ee5f8} stack=[0xc000060000,0xc000060800)
fatal error: unknown caller pc
runtime stack:
runtime.throw(0x131866ca2, 0x11)
runtime/panic.go:1116 +0x68
runtime.gentraceback(0x130e49434, 0x3fff6f7ee5b8, 0x0, 0xc000602900, 0x0, 0x0, 0x7fffffff, 0x3fff6f7ee608, 0x0, 0x0, …)
runtime/traceback.go:273 +0x15dc
runtime.addOneOpenDeferFrame.func1()
runtime/panic.go:721 +0xa0
runtime.systemstack(0x0)
runtime/asm_ppc64x.s:269 +0xa0
runtime.mstart()
runtime/proc.go:1116
goroutine 14 [running]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0x0
stack: frame={sp:0x3fff6f7ee468, fp:0x3fff6f7ee488} stack=[0xc000060000,0xc000060800)
runtime.systemstack_switch()
runtime/asm_ppc64x.s:211 +0x10 fp=0x3fff6f7ee488 sp=0x3fff6f7ee468 pc=0x130e66b60
created by text/template/parse.lex
text/template/parse/lex.go:223 +0x1b4
goroutine 1 [runnable]:
time.loadLocation(0x13185c26a, 0x9, 0xc00035ecb0, 0x1, 0x1, 0x18, 0x3fff6d752d00, 0x3fff6d752d00)
time/zoneinfo_read.go:519 +0x24
time.initLocal()
time/zoneinfo_unix.go:37 +0x1e4
sync.(*Once).doSlow(0x1322ec1e8, 0x131c46610)
sync/once.go:66 +0x16c
sync.(*Once).Do(…)
sync/once.go:57
time.(*Location).get(0x1322b52e0, 0x131c992e0)
time/zoneinfo.go:92 +0xac
time.Time.locabs(0xbfeeb4d922185643, 0x11a9e84, 0x1322b52e0, 0x10, 0x98, 0x97, 0x131a9da20)
time/time.go:455 +0x64
time.Time.AppendFormat(0xbfeeb4d922185643, 0x11a9e84, 0x1322b52e0, 0xc00035f020, 0x0, 0x40, 0x3fff6d55179d, 0xf, 0x3fff6d752d00, 0x208b884f587858, …)
time/format.go:513 +0x60
time.Time.Format(0xbfeeb4d922185643, 0x11a9e84, 0x1322b52e0, 0x3fff6d55179d, 0xf, 0xc0001b0180, 0xc00060d100)
time/format.go:505 +0xb4
log/syslog.(*netConn).writeString(0xc0004fd2c0, 0x5, 0xc0007a66f0, 0x8, 0x3fffce3bf5f6, 0xb, 0xc00060d200, 0xf4, 0x3fff6d54eac2, 0x1, …)
log/syslog/syslog.go:288 +0x74
log/syslog.(*Writer).write(0xc0004a2180, 0x5, 0xc00060d200, 0xf4, 0x3fff6d549874, 0x131a9da20, 0xc000507268)
log/syslog/syslog.go:273 +0xd8
log/syslog.(*Writer).writeAndRetry(0xc0004a2180, 0x5, 0xc00060d200, 0xf4, 0x0, 0x0, 0x0)
log/syslog/syslog.go:254 +0x1a4
log/syslog.(*Writer).Write(0xc0004a2180, 0xc0001b0180, 0xf4, 0x180, 0x131c99201, 0x131a9da20, 0xc000507268)
log/syslog/syslog.go:175 +0x8c
fmt.Fprint(0x3fff6d6284a0, 0xc0004a2180, 0xc00035f428, 0x1, 0x1, 0x1, 0xc0004fd2c0, 0x3fff6d5495b8)
fmt/print.go:233 +0xa4
text/template.(*state).printValue(0xc00035f618, 0x131c8d2e0, 0xc0002f94d0, 0x131a9da20, 0xc000507268, 0x98)
text/template/exec.go:960 +0xdc
text/template.(*state).walk(0xc00035f618, 0x3fff6d6145a0, 0xc0002b5520, 0x99, 0x131c8d2e0, 0xc0002f94d0)
text/template/exec.go:257 +0x368
text/template.(*state).walk(0xc00035f618, 0x3fff6d6145a0, 0xc0002b5520, 0x99, 0x131c8d640, 0xc0002f91a0)
text/template/exec.go:263 +0x12c
text/template.(*Template).execute(0xc000507100, 0x3fff6d6284a0, 0xc0004a2180, 0x3fff6d6145a0, 0xc0002b5520, 0x0, 0x0)
text/template/exec.go:220 +0x198
text/template.(*Template).Execute(…)
text/template/exec.go:203
github.com/sylabs/singularity/pro-log-plugin.SyslogOutput.Log(0xc0007a6650, 0xd, 0xc0007a6670, 0x6, 0xc0007a6690, 0xb, 0xc00015f680, 0x3b, 0xc000507100, 0x3ec, …)
github.com/sylabs/singularity/pro-log-plugin/syslog.go:61 +0x17c
github.com/sylabs/singularity/pro-log-plugin.(*pluginImplementation).callOutputs(0xc000131b00)
github.com/sylabs/singularity/pro-log-plugin/main.go:79 +0x3f8
github.com/sylabs/singularity/pro-log-plugin.callbackLog.func1(0x13224dd80, 0x1322ebfc8, 0x0, 0x0, 0x0, 0x0)
github.com/sylabs/singularity/pro-log-plugin/main.go:60 +0x164
github.com/spf13/cobra.(*Command).execute(0x13224dd80, 0xc00000e070, 0x0, 0x0, 0x13224dd80, 0xc00000e070)
github.com/spf13/cobra@v1.1.1/command.go:829 +0x4ac
github.com/spf13/cobra.(*Command).ExecuteC(0x13224dae0, 0x131c441f8, 0xc000097da0, 0xc0002d9550)
github.com/spf13/cobra@v1.1.1/command.go:958 +0x2b4
github.com/spf13/cobra.(*Command).Execute(…)
github.com/spf13/cobra@v1.1.1/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(…)
github.com/spf13/cobra@v1.1.1/command.go:888
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity()
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:431 +0x1b0
main.main()
github.com/sylabs/singularity@v0.0.0/cmd/singularity/cli.go:19 +0x64
goroutine 12 [syscall]:
os/signal.signal_recv(0x0)
runtime/sigqueue.go:147 +0x10c
os/signal.loop()
os/signal/signal_unix.go:23 +0x38
created by os/signal.Notify.func1.1
os/signal/signal.go:150 +0x60
goroutine 13 [select]:
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity.func2(0xc000097da0, 0xc0002d9550, 0x131c85ae0, 0xc000506f80)
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:423 +0xc0
created by github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:422 +0x178

Backtrace 3

runtime: unexpected return pc for runtime.sigpanic called from 0x3fff6180e5b8
stack: frame={sp:0x3fff6180e5b8, fp:0x3fff6180e5f8} stack=[0xc0004f5800,0xc0004f6000)
fatal error: unknown caller pc
runtime stack:
runtime.throw(0x137346ca2, 0x11)
runtime/panic.go:1116 +0x68
runtime.gentraceback(0x136929434, 0x3fff6180e5b8, 0x0, 0xc00008a780, 0x0, 0x0, 0x7fffffff, 0x3fff6180e608, 0x0, 0x0, …)
runtime/traceback.go:273 +0x15dc
runtime.addOneOpenDeferFrame.func1()
runtime/panic.go:721 +0xa0
runtime.systemstack(0x0)
runtime/asm_ppc64x.s:269 +0xa0
runtime.mstart()
runtime/proc.go:1116
goroutine 52 [running]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0x0
stack: frame={sp:0x3fff6180e468, fp:0x3fff6180e488} stack=[0xc0004f5800,0xc0004f6000)
runtime.systemstack_switch()
runtime/asm_ppc64x.s:211 +0x10 fp=0x3fff6180e488 sp=0x3fff6180e468 pc=0x136946b60
created by text/template/parse.lex
text/template/parse/lex.go:223 +0x1b4
goroutine 1 [runnable]:
fmt.Fprintf(0x3fff5bfc0008, 0xc000208108, 0x3fff604428f8, 0x13, 0xc00035f100, 0x6, 0x6, 0xf4, 0xe, 0x3fff602d802c)
fmt/print.go:202 +0x2c
log/syslog.(*netConn).writeString(0xc000204ae0, 0x5, 0xc0002708f0, 0x8, 0x3fffd579f5f6, 0xb, 0xc000094200, 0xf4, 0x3fff6043eac2, 0x1, …)
log/syslog/syslog.go:289 +0x230
log/syslog.(*Writer).write(0xc0003622a0, 0x5, 0xc000094200, 0xf4, 0x3fff60439874, 0x13757da20, 0xc0000a0368)
log/syslog/syslog.go:273 +0xd8
log/syslog.(*Writer).writeAndRetry(0xc0003622a0, 0x5, 0xc000094200, 0xf4, 0x0, 0x0, 0x0)
log/syslog/syslog.go:254 +0x1a4
log/syslog.(*Writer).Write(0xc0003622a0, 0xc00053c000, 0xf4, 0x140, 0x137779201, 0x13757da20, 0xc0000a0368)
log/syslog/syslog.go:175 +0x8c
fmt.Fprint(0x3fff605184a0, 0xc0003622a0, 0xc00035f428, 0x1, 0x1, 0x1, 0xc000204ae0, 0x3fff604395b8)
fmt/print.go:233 +0xa4
text/template.(*state).printValue(0xc00035f618, 0x13776d2e0, 0xc0001956b0, 0x13757da20, 0xc0000a0368, 0x98)
text/template/exec.go:960 +0xdc
text/template.(*state).walk(0xc00035f618, 0x3fff605045a0, 0xc000522750, 0x99, 0x13776d2e0, 0xc0001956b0)
text/template/exec.go:257 +0x368
text/template.(*state).walk(0xc00035f618, 0x3fff605045a0, 0xc000522750, 0x99, 0x13776d640, 0xc0001951d0)
text/template/exec.go:263 +0x12c
text/template.(*Template).execute(0xc0000a0200, 0x3fff605184a0, 0xc0003622a0, 0x3fff605045a0, 0xc000522750, 0x0, 0x0)
text/template/exec.go:220 +0x198
text/template.(*Template).Execute(…)
text/template/exec.go:203
github.com/sylabs/singularity/pro-log-plugin.SyslogOutput.Log(0xc000270800, 0xd, 0xc000270840, 0x6, 0xc000270870, 0xb, 0xc000228000, 0x3b, 0xc0000a0200, 0x3ec, …)
github.com/sylabs/singularity/pro-log-plugin/syslog.go:61 +0x17c
github.com/sylabs/singularity/pro-log-plugin.(*pluginImplementation).callOutputs(0xc000230480)
github.com/sylabs/singularity/pro-log-plugin/main.go:79 +0x3f8
github.com/sylabs/singularity/pro-log-plugin.callbackLog.func1(0x137d2dd80, 0x137dcbfc8, 0x0, 0x0, 0x0, 0x0)
github.com/sylabs/singularity/pro-log-plugin/main.go:60 +0x164
github.com/spf13/cobra.(*Command).execute(0x137d2dd80, 0xc00000e070, 0x0, 0x0, 0x137d2dd80, 0xc00000e070)
github.com/spf13/cobra@v1.1.1/command.go:829 +0x4ac
github.com/spf13/cobra.(*Command).ExecuteC(0x137d2dae0, 0x1377241f8, 0xc0004d60c0, 0xc0002dbee0)
github.com/spf13/cobra@v1.1.1/command.go:958 +0x2b4
github.com/spf13/cobra.(*Command).Execute(…)
github.com/spf13/cobra@v1.1.1/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(…)
github.com/spf13/cobra@v1.1.1/command.go:888
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity()
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:431 +0x1b0
main.main()
github.com/sylabs/singularity@v0.0.0/cmd/singularity/cli.go:19 +0x64
goroutine 50 [syscall]:
os/signal.signal_recv(0x0)
runtime/sigqueue.go:147 +0x10c
os/signal.loop()
os/signal/signal_unix.go:23 +0x38
created by os/signal.Notify.func1.1
os/signal/signal.go:150 +0x60
goroutine 51 [select]:
github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity.func2(0xc0004d60c0, 0xc0002dbee0, 0x137765ae0, 0xc0001cbb00)
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:423 +0xc0
created by github.com/sylabs/singularity/cmd/internal/cli.ExecuteSingularity
github.com/sylabs/singularity@v0.0.0/cmd/internal/cli/singularity.go:422 +0x178

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (14 by maintainers)

Commits related to this issue

misc/cgo/testplugin: skip TestIssue25756pie on darwin/arm64 builder This test is known to be broken on the darwin/arm64 builder. Skip it while it's being investigated so it doesn't mask other failure... — committed to golang/go by dmitshur 3 years ago

Most upvoted comments

This does fix the issue. Using the patched go, compiled application and plugin with -buildmode=pie on the ppc64le system and it works as expected. Many thanks for you efforts on this @laboger!

dtrudg on May 12, 2021

I think I am able to reproduce this problem with a test where the main program is built with pie and it invokes a plugin. At some point after it returns from the plugin the value of r2 is incorrect which can lead to various errors. I am still trying to debug it to find where it is happening. This failure won’t occur if the main program is built with -buildmode=default, because in that case the main program doesn’t depend on the r2 value.

laboger on Dec 21, 2020