go: regexp: unexpected return pc for regexp.(*Regexp).tryBacktrack called
What version of Go are you using (go version)?
$ go version go version go1.19rc2 linux/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go envGO111MODULE=“” GOARCH=“amd64” GOBIN=“” GOEXE=“” GOEXPERIMENT=“” GOFLAGS=“” GOHOSTARCH=“amd64” GOHOSTOS=“linux” GOINSECURE=“” GONOPROXY=“” GONOSUMDB=“” GOOS=“linux” GOPRIVATE=“” GOPROXY=“https://proxy.golang.org,direct” GOSUMDB=“sum.golang.org” GOTMPDIR=“” GOVCS=“” GOVERSION=“go1.19rc2” GCCGO=“gccgo” GOAMD64=“v1” AR=“ar” CC=“gcc” CXX=“g++” CGO_ENABLED=“1” GOMOD=“/dev/null” GOWORK=“” CGO_CFLAGS=“-g -O2” CGO_CPPFLAGS=“” CGO_CXXFLAGS=“-g -O2” CGO_FFLAGS=“-g -O2” CGO_LDFLAGS=“-g -O2” PKG_CONFIG=“pkg-config” GOGCCFLAGS=“-fPIC -m64 -pthread -Wl,–no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1385535929=/tmp/go-build -gno-record-gcc-switches”
What did you do?
Encountered a runtime crash on 1.19rc2 in the regexp package when running a configuration that uses plugins. The specific configuration was https://github.com/golangci/golangci-lint combined with a custom linter plugin which happened to use a regexp internally. The original issue was https://github.com/kubernetes/kubernetes/issues/111297
The following is a minimal reproducer which is compiled as a plugin and then run:
plugin.go:
// To reproduce, run:
//
// go build -buildmode=plugin ./plugin.go
// go run ./plugin.go
package main
import (
"context"
"fmt"
"plugin"
"regexp"
"time"
)
type EvaluatorInterface interface {
Evaluate()
EvaluateLoop(context.Context)
}
type EvaluatorImpl struct{}
func (EvaluatorImpl) Evaluate() {
R.MatchString("123")
}
func (EvaluatorImpl) EvaluateLoop(ctx context.Context) {
for ctx.Err() == nil {
R.MatchString("123")
}
}
var Evaluator EvaluatorImpl
var R = regexp.MustCompile(".*")
func main() {
ctx, cancel := context.WithCancel(context.Background())
// works
go mainLoop(ctx)
time.Sleep(time.Second)
// breaks
go pluginLoop(ctx)
time.Sleep(time.Second)
// breaks
go mainLoopPluginEval(ctx)
time.Sleep(time.Second)
cancel()
}
func getEvaluator() EvaluatorInterface {
p, err := plugin.Open("plugin.so")
if err != nil {
panic(err)
}
evaluatorInterface, err := p.Lookup("Evaluator")
if err != nil {
panic(err)
}
evaluator, ok := evaluatorInterface.(EvaluatorInterface)
if !ok {
panic("doesn't implement EvaluatorInterface")
}
return evaluator
}
// loops and evaluates inside the main binary
func mainLoop(ctx context.Context) {
fmt.Println("mainLoop")
for ctx.Err() == nil {
R.MatchString("123")
}
}
// loops and evaluates inside the plugin
func pluginLoop(ctx context.Context) {
fmt.Println("pluginLoop")
getEvaluator().EvaluateLoop(ctx)
}
// loops inside the main binary, evaluates inside the plugin
func mainLoopPluginEval(ctx context.Context) {
fmt.Println("mainLoopPluginEval")
evaluator := getEvaluator()
for ctx.Err() == nil {
evaluator.Evaluate()
}
}
Running that on 1.19rc2 produces:
$ go version
go version go1.19rc2 linux/amd64
$ go build -buildmode=plugin ./plugin.go
$ go run ./plugin.go
mainLoop
pluginLoop
mainLoop
pluginLoop
runtime: g 3: unexpected return pc for regexp.(*Regexp).tryBacktrack called from 0xc00012e000
stack: frame={sp:0xc000129d08, fp:0xc000129dc0} stack=[0xc000128000,0xc00012a000)
0x000000c000129c08: 0x000000c000129c10 0x00000000004aabe0 <runtime.addOneOpenDeferFrame.func1+0x0000000000000000>
0x000000c000129c18: 0x0000000000560005 <regexp.(*Regexp).tryBacktrack+0x0000000000000cc5> 0x000000c000129d08
0x000000c000129c28: 0x000000c000007380 0x0000000000000000
0x000000c000129c38: 0x000000c000129cf8 0x00000000004ab3d2 <runtime.gopanic+0x0000000000000112>
0x000000c000129c48: 0x000000c000129c88 0x00007f9c5d8cf2e5 <runtime.(*mcache).refill+0x00000000000001e5>
0x000000c000129c58: 0x00000000006cc9e0 0x00007f9c5d8daaa7 <runtime.(*gcControllerState).heapGoalInternal+0x0000000000000027>
0x000000c000129c68: 0x00007f9c877e2290 0x0000000000400000
0x000000c000129c78: 0x000000c000129cb8 0x00007f9c5d8dabe5 <runtime.(*gcControllerState).trigger+0x0000000000000025>
0x000000c000129c88: 0x00000000006cd820 0x00007f9c5d8c51be <runtime.(*mcache).nextFree+0x000000000000009e>
0x000000c000129c98: 0x00007f9c877e2180 0x0000000000400000
0x000000c000129ca8: 0x000000c000129ce8 0x000000c0000073a0
0x000000c000129cb8: 0x000000c000007380 0x0000000000000000
0x000000c000129cc8: 0x000000000057d2a0 0x00000000005ce300
0x000000c000129cd8: 0x0000000000000000 0x0000000000000000
0x000000c000129ce8: 0x0000000000000000 0x0000000000000000
0x000000c000129cf8: 0x000000c000129da0 0x0000000000560005 <regexp.(*Regexp).tryBacktrack+0x0000000000000cc5>
0x000000c000129d08: <0x000000000057d2a0 0x00000000005ce300
0x000000c000129d18: 0x00007f9c877d7108 0x0000000000008000
0x000000c000129d28: 0x000000c000058c00 0x000000c000132000
0x000000c000129d38: 0x0000000000000000 0x00000000000000a0
0x000000c000129d48: 0x000000c000009d70 0x00007f9c00000000
0x000000c000129d58: 0x0000000000008000 0x000000000057d3a0
0x000000c000129d68: 0x0000000000000001 0x0000000000000000
0x000000c000129d78: 0x00007f9c5d9a14d3 <regexp.(*bitState).reset+0x0000000000000113> 0x000000000000000a
0x000000c000129d88: 0x0000000000000000 0x000000c0000000a0
0x000000c000129d98: 0x000000c0000000a0 0x000000c0000000a0
0x000000c000129da8: 0x00007f9c5d9a290d <regexp.(*Regexp).backtrack+0x000000000000032d> 0x000000c000000140
0x000000c000129db8: !0x000000c00012e000 >0x00000000005ceb88
0x000000c000129dc8: 0x000000c00012e080 0x000000c000129e80
0x000000c000129dd8: 0x000000000047fd9d <runtime.(*itab).init+0x000000000000019d> 0x000000c00007e090
0x000000c000129de8: 0x0000000000000003 0x00000000005ceb88
0x000000c000129df8: 0x0000000000000002 0x0000000000000000
0x000000c000129e08: 0x000000c00012e080 0x000000c00012e000
0x000000c000129e18: 0x000000000000000c 0x000000c000129ed0
0x000000c000129e28: 0x00007f9c5d9a5470 <regexp.(*Regexp).doExecute+0x0000000000000290> 0x00000000006ccd00
0x000000c000129e38: 0x0000000000000000 0x0000000000000000
0x000000c000129e48: 0x000000c000000140 0x0000000000000000
0x000000c000129e58: 0x0000000000000000 0x0000000000000000
0x000000c000129e68: 0x00007f9c5d9b10d4 0x0000000000000003
0x000000c000129e78: 0x0000000000000000 0x000000c000129ec8
0x000000c000129e88: 0x000000000047f853 <runtime.getitab+0x0000000000000153> 0x00007f9c5fb5b7d0
0x000000c000129e98: 0x0000000000000000 0x0000000000000000
0x000000c000129ea8: 0x0000000000000000 0x000000c00007e060
0x000000c000129eb8: 0x000000c00011e000
fatal error: unknown caller pc
runtime stack:
runtime.throw({0x5a1909?, 0x6911a0?})
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/panic.go:1047 +0x5d fp=0x7f9c5f3529e8 sp=0x7f9c5f3529b8 pc=0x4ababd
runtime.gentraceback(0x7f9c877e21e3?, 0x8?, 0xc000034000?, 0x7f9c5f352dc0?, 0x0, 0x0, 0x7fffffff, 0x7f9c5f352da8, 0x28?, 0x0)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/traceback.go:258 +0x1cf7 fp=0x7f9c5f352d58 sp=0x7f9c5f3529e8 pc=0x4d0197
runtime.addOneOpenDeferFrame.func1()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/panic.go:645 +0x6b fp=0x7f9c5f352dd0 sp=0x7f9c5f352d58 pc=0x4aac4b
runtime.systemstack()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:492 +0x49 fp=0x7f9c5f352dd8 sp=0x7f9c5f352dd0 pc=0x4d9de9
goroutine 3 [running]:
runtime.systemstack_switch()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:459 fp=0xc000129c08 sp=0xc000129c00 pc=0x4d9d80
runtime.addOneOpenDeferFrame(0xc000129c88?, 0x7f9c5d8cf2e5?, 0x6cc9e0?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/panic.go:644 +0x69 fp=0xc000129c48 sp=0xc000129c08 pc=0x4aab89
panic({0x57d2a0, 0x5ce300})
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/panic.go:844 +0x112 fp=0xc000129d08 sp=0xc000129c48 pc=0x4ab3d2
runtime: g 3: unexpected return pc for regexp.(*Regexp).tryBacktrack called from 0xc00012e000
stack: frame={sp:0xc000129d08, fp:0xc000129dc0} stack=[0xc000128000,0xc00012a000)
0x000000c000129c08: 0x000000c000129c10 0x00000000004aabe0 <runtime.addOneOpenDeferFrame.func1+0x0000000000000000>
0x000000c000129c18: 0x0000000000560005 <regexp.(*Regexp).tryBacktrack+0x0000000000000cc5> 0x000000c000129d08
0x000000c000129c28: 0x000000c000007380 0x0000000000000000
0x000000c000129c38: 0x000000c000129cf8 0x00000000004ab3d2 <runtime.gopanic+0x0000000000000112>
0x000000c000129c48: 0x000000c000129c88 0x00007f9c5d8cf2e5 <runtime.(*mcache).refill+0x00000000000001e5>
0x000000c000129c58: 0x00000000006cc9e0 0x00007f9c5d8daaa7 <runtime.(*gcControllerState).heapGoalInternal+0x0000000000000027>
0x000000c000129c68: 0x00007f9c877e2290 0x0000000000400000
0x000000c000129c78: 0x000000c000129cb8 0x00007f9c5d8dabe5 <runtime.(*gcControllerState).trigger+0x0000000000000025>
0x000000c000129c88: 0x00000000006cd820 0x00007f9c5d8c51be <runtime.(*mcache).nextFree+0x000000000000009e>
0x000000c000129c98: 0x00007f9c877e2180 0x0000000000400000
0x000000c000129ca8: 0x000000c000129ce8 0x000000c0000073a0
0x000000c000129cb8: 0x000000c000007380 0x0000000000000000
0x000000c000129cc8: 0x000000000057d2a0 0x00000000005ce300
0x000000c000129cd8: 0x0000000000000000 0x0000000000000000
0x000000c000129ce8: 0x0000000000000000 0x0000000000000000
0x000000c000129cf8: 0x000000c000129da0 0x0000000000560005 <regexp.(*Regexp).tryBacktrack+0x0000000000000cc5>
0x000000c000129d08: <0x000000000057d2a0 0x00000000005ce300
0x000000c000129d18: 0x00007f9c877d7108 0x0000000000008000
0x000000c000129d28: 0x000000c000058c00 0x000000c000132000
0x000000c000129d38: 0x0000000000000000 0x00000000000000a0
0x000000c000129d48: 0x000000c000009d70 0x00007f9c00000000
0x000000c000129d58: 0x0000000000008000 0x000000000057d3a0
0x000000c000129d68: 0x0000000000000001 0x0000000000000000
0x000000c000129d78: 0x00007f9c5d9a14d3 <regexp.(*bitState).reset+0x0000000000000113> 0x000000000000000a
0x000000c000129d88: 0x0000000000000000 0x000000c0000000a0
0x000000c000129d98: 0x000000c0000000a0 0x000000c0000000a0
0x000000c000129da8: 0x00007f9c5d9a290d <regexp.(*Regexp).backtrack+0x000000000000032d> 0x000000c000000140
0x000000c000129db8: !0x000000c00012e000 >0x00000000005ceb88
0x000000c000129dc8: 0x000000c00012e080 0x000000c000129e80
0x000000c000129dd8: 0x000000000047fd9d <runtime.(*itab).init+0x000000000000019d> 0x000000c00007e090
0x000000c000129de8: 0x0000000000000003 0x00000000005ceb88
0x000000c000129df8: 0x0000000000000002 0x0000000000000000
0x000000c000129e08: 0x000000c00012e080 0x000000c00012e000
0x000000c000129e18: 0x000000000000000c 0x000000c000129ed0
0x000000c000129e28: 0x00007f9c5d9a5470 <regexp.(*Regexp).doExecute+0x0000000000000290> 0x00000000006ccd00
0x000000c000129e38: 0x0000000000000000 0x0000000000000000
0x000000c000129e48: 0x000000c000000140 0x0000000000000000
0x000000c000129e58: 0x0000000000000000 0x0000000000000000
0x000000c000129e68: 0x00007f9c5d9b10d4 0x0000000000000003
0x000000c000129e78: 0x0000000000000000 0x000000c000129ec8
0x000000c000129e88: 0x000000000047f853 <runtime.getitab+0x0000000000000153> 0x00007f9c5fb5b7d0
0x000000c000129e98: 0x0000000000000000 0x0000000000000000
0x000000c000129ea8: 0x0000000000000000 0x000000c00007e060
0x000000c000129eb8: 0x000000c00011e000
regexp.(*Regexp).tryBacktrack(0x5ceb88, 0xc00012e080, {0xc000129e80?, 0x47fd9d}, 0x7e090?, 0x3?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/regexp/backtrack.go:172 +0xcc5 fp=0xc000129dc0 sp=0xc000129d08 pc=0x560005
created by main.main
/usr/local/google/home/liggitt/tmp/119regex/plugin.go:42 +0xd5
goroutine 1 [sleep]:
runtime.gopark(0x33d97f6c6b64c?, 0xc000062f18?, 0xc0?, 0x6c?, 0xc000012018?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:363 +0xd6 fp=0xc000062f08 sp=0xc000062ee8 pc=0x4ae9f6
time.Sleep(0x3b9aca00)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/time.go:195 +0x135 fp=0xc000062f48 sp=0xc000062f08 pc=0x4d8e55
main.main()
/usr/local/google/home/liggitt/tmp/119regex/plugin.go:43 +0xdf fp=0xc000062f80 sp=0xc000062f48 pc=0x56cb9f
runtime.main()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:250 +0x212 fp=0xc000062fe0 sp=0xc000062f80 pc=0x4ae632
runtime.goexit()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x4dbfa1
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:363 +0xd6 fp=0xc000054fb0 sp=0xc000054f90 pc=0x4ae9f6
runtime.goparkunlock(...)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:369
runtime.forcegchelper()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:302 +0xad fp=0xc000054fe0 sp=0xc000054fb0 pc=0x4ae88d
runtime.goexit()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000054fe8 sp=0xc000054fe0 pc=0x4dbfa1
created by runtime.init.6
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:290 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:363 +0xd6 fp=0xc000050790 sp=0xc000050770 pc=0x4ae9f6
runtime.goparkunlock(...)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgcsweep.go:278 +0x8e fp=0xc0000507c8 sp=0xc000050790 pc=0x499cae
runtime.gcenable.func1()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgc.go:178 +0x26 fp=0xc0000507e0 sp=0xc0000507c8 pc=0x48e8a6
runtime.goexit()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000507e8 sp=0xc0000507e0 pc=0x4dbfa1
created by runtime.gcenable
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgc.go:178 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc000096000?, 0x5cd960?, 0x1?, 0x0?, 0x0?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:363 +0xd6 fp=0xc000050f70 sp=0xc000050f50 pc=0x4ae9f6
runtime.goparkunlock(...)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x69e280)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000050fa0 sp=0xc000050f70 pc=0x497cd3
runtime.bgscavenge(0x0?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgcscavenge.go:617 +0x45 fp=0xc000050fc8 sp=0xc000050fa0 pc=0x4982c5
runtime.gcenable.func2()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgc.go:179 +0x26 fp=0xc000050fe0 sp=0xc000050fc8 pc=0x48e846
runtime.goexit()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000050fe8 sp=0xc000050fe0 pc=0x4dbfa1
created by runtime.gcenable
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mgc.go:179 +0xaa
goroutine 20 [finalizer wait]:
runtime.gopark(0x69e760?, 0xc00008a820?, 0x0?, 0x0?, 0xc000054770?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:363 +0xd6 fp=0xc000054628 sp=0xc000054608 pc=0x4ae9f6
runtime.goparkunlock(...)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/proc.go:369
runtime.runfinq()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mfinal.go:180 +0x10f fp=0xc0000547e0 sp=0xc000054628 pc=0x48d9af
runtime.goexit()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000547e8 sp=0xc0000547e0 pc=0x4dbfa1
created by runtime.createfing
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/mfinal.go:157 +0x45
goroutine 21 [runnable]:
context.(*cancelCtx).Err(0xc00009a040?)
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/context/context.go:373 +0x9b fp=0xc000063f38 sp=0xc000063f30 pc=0x50009b
main.mainLoop({0x5ce9b8, 0xc00009a040})
/usr/local/google/home/liggitt/tmp/119regex/plugin.go:71 +0xb4 fp=0xc000063fc0 sp=0xc000063f38 pc=0x56cf54
main.main.func1()
/usr/local/google/home/liggitt/tmp/119regex/plugin.go:38 +0x2a fp=0xc000063fe0 sp=0xc000063fc0 pc=0x56cd0a
runtime.goexit()
/usr/local/google/home/liggitt/.gvm/gos/go1.19rc2/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x4dbfa1
created by main.main
/usr/local/google/home/liggitt/tmp/119regex/plugin.go:38 +0x85
exit status 2
Running that on go1.18.4 works fine:
$ go version
go version go1.18.4 linux/amd64
$ go build -buildmode=plugin ./plugin.go
$ go run ./plugin.go
mainLoop
pluginLoop
mainLoopPluginEval
What did you expect to see?
Regexp calls inside a plugin work.
What did you see instead?
Regexp calls inside a plugin trigger runtime errors.
@dims found https://github.com/golang/go/commit/0293c51bc5d8ca0728913c4b7f9f92339f8fd9a6, which looks related. Reverting that fixed the issue in the linter scenario we encountered, but did not resolve the issue with this reproducer edit: reverting 0293c51bc5d8ca0728913c4b7f9f92339f8fd9a6 did not fix the linter scenario, we were mistaken about that
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (12 by maintainers)
Yes, this bug depends on compiler and architecture details, and also how the dynamic linker resolves symbols, so it is very system dependent.
I think I understand the problem, will send a CL soon. Thanks.
I don’t have access to do so, but feel free to close https://github.com/golang/go/issues/54010
We don’t need backport. Go 1.19 hasn’t been released, and the tree is not open for 1.20 development. Go 1.19 will be cut on the main branch, which includes the CL above.
@cherrymui sweeeeetttt! the change https://go.dev/cl/418960 fixed the original problem we had in https://github.com/kubernetes/kubernetes/issues/111297
The plugin needs to be position independent code, whereas the main executable doesn’t. This causes slight differences in code generation, e.g. the use of GOT, TLS model, etc.