go: runtime: execution halts with goroutines stuck in `runtime.gopark` and `runtime.futex`

What version of Go are you using (`go version`)?

go version go1.20.1 linux/amd64

Does this issue reproduce with the latest release?

Yes, with 1.20.1

What operating system and processor architecture are you using (`go env`)?

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

What did you do?

After upgrading avalanchego to use go1.20.1 instead of 1.19.6, the binary now randomly halts (all goroutines stop and the program becomes unresponsive) after about 20-30 minutes of runtime on go1.20.1 linux/amd64.

Notably, the binary does not panic or exit at this point (as you would expect if it was deadlocked). It just stays pegged to ~30% CPU and spins indefinitely on something (but all async processes halt):

The output of grs -r on dlv is attached from the point of stall: goroutines.txt

What did you expect to see?

The program to work like it did on v1.19.6 (not halt without warning or panic). To run this test (and hopefully avoid wasting your time), we compiled the exact same code in both 1.19.6 and 1.20.1 (this only occurred in 1.20.1).

What did you see instead?

The program halted without emitting a panic or any other info that could be used to diagnose the issue. All metrics-related goroutines that would normally inform of system level processes also stopped running (I.e. global program halt).

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (5 by maintainers)

Most upvoted comments

@manav2401 There are a few options:

Build directly:

$ git clone https://go.googlesource.com/go
$ cd go/src
$ git checkout release-branch.go1.20
$ ./make.bash
$ # Now build with `../bin/go build`.

Use gotip:

$ go install golang.org/dl/gotip@latest
$ gotip download release-branch.go1.20
$ # Now build with `gotip build ...`

prattmic on Mar 3, 2023

That’s great!

The issue is with traceback generally. Traceback is used in panic processing, but also in other places. In your case, it is stuck in traceback during CPU profiling.

I’m going to close this as a duplicate of https://github.com/golang/go/issues/58513. Please reopen if the issue does reproduce after all.

prattmic on Mar 1, 2023

@patrick-ogrady go1.20.2Not yet released according to https://github.com/golang/go/issues/58513#issuecomment -1430375549, the master branch contains patches, which can be accessed through go install golang.org/dl/ gotip@latest gotip download Then use the latest tool chain through gotip

qiulaidongfeng on Mar 1, 2023