go: runtime: execution halts with goroutines stuck in `runtime.gopark` (`protocol error E08 during memory read for packet`)
What version of Go are you using (go version)?
go version go1.20.7 darwin/arm64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
GOARCH="arm64"
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
What did you do?
When running continuous profiling on my binary, the entire program halted with all goroutines stuck in runtime.gopark. When disabling profiling, this problem went away.
I posted a similar issue a few months ago (“halt when profiling”) but don’t believe it to be related (https://github.com/golang/go/issues/58798).
Goroutine 1 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [semacquire 258316338387791]
Goroutine 2 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4)
Goroutine 4 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 5 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call 258796780404375]
Goroutine 6 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call 258856782846166]
Goroutine 7 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call 258856782846166]
Goroutine 8 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 9 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 10 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [chan receive]
Goroutine 18 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [GC sweep wait]
Goroutine 19 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [GC scavenge wait]
Goroutine 20 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 21 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 22 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 23 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 26 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [chan receive 258316338387791]
Goroutine 33 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258856782846166]
Goroutine 34 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [finalizer wait]
Goroutine 36 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call 258316186592916]
Goroutine 37 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [debug call]
Goroutine 44 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258316338387791]
Goroutine 52 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [chan receive 258316338387791]
Goroutine 54 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258316338387791]
Goroutine 58 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [IO wait 258316338387791]
Goroutine 59 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [IO wait 258316338387791]
Goroutine 61 - Runtime: :0 ??? (0x1ab4ccacc) (thread 7681484)
Goroutine 62 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [chan receive 258316338387791]
Goroutine 66 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258316338387791]
Goroutine 67 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 68 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258316338387791]
Goroutine 69 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258316338387791]
Goroutine 88 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 340 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [chan receive]
Goroutine 347 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258362045239291]
Goroutine 607 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select 258376783287916]
Goroutine 619 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 620 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 621 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 622 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [semacquire]
Goroutine 632 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 633 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [IO wait]
Goroutine 634 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [select]
Goroutine 635 - Runtime: /usr/local/go/src/runtime/proc.go:382 runtime.gopark (0x102255ab4) [chan receive 258316810183875]
What did you expect to see?
I expected profiling to allow the binary to run without issue.
What did you see instead?
Profiling put the binary in a “deadlocked”/“unrecoverable” state.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 27 (12 by maintainers)
Commits related to this issue
- runtime: guard against runtime/sema* ops on Darwin signal stack. These operations misbehave and cause hangs and flakes. Fail hard if they are attempted. Tested by backing out the Darwin-profiling-ha... — committed to golang/go by dr2chase a year ago
- [release-branch.go1.20] runtime: profiling on Darwin cannot use blocking reads On Darwin (and assume also on iOS but not sure), notetsleepg cannot be called in a signal-handling context. Avoid this ... — committed to golang/go by dr2chase a year ago
- [release-branch.go1.21] runtime: profiling on Darwin cannot use blocking reads On Darwin (and assume also on iOS but not sure), notetsleepg cannot be called in a signal-handling context. Avoid this ... — committed to golang/go by dr2chase a year ago
definitely have a fix, maybe it’s not the right fix.
Actually got it open just now, would not want to claim it is anything like “progress”. Currently cursing at lldb, and about to double-check Go’s signal trampoline code.