go: testing: provide a way to work around "race: limit on 8128 simultaneously alive goroutines is exceeded" error
$ go version go version devel go1.17-dc00dc6c6b Thu Jun 10 12:41:37 2021 +0000 linux/amd64
Issue #38184 seems to be a reasonably hard limit in the race detector library, but it’s not always easy to work around it.
In tests, particularly, this issue more commonly arises in higher level tests that run more slowly and hence use testing.T.Parallel
to speed themselves up. But this means that if a large instance is used to run tests (for example in continuous integration), we might see more simultaneously alive goroutines because the value of GOMAXPROCS
is greater so more tests will be run in parallel, leading to this issue arising.
There are a few current possibilities for working around this at present, although none of them feel great:
- avoid calling
testing.T.Parallel
when the race detector is enabled, but this disables parallelism completely, which feels like overkill. - run some packages with a different value for the
-test.parallel
flag, but this means that the top level CI script needs to be changed (you’ll no longer be able to rungo test -race ./...
) which has maintainability issues. - update the value of
-test.parallel
within a given package by usingflag.Lookup
and setting the flag directly, but this is a nasty hack.
Ideally, we’d be able to configure the maximum number of active goroutines allowed, but failing that, another possibility might be to make the default value of testing.parallel
come from runtime.GOMAXPROCS
after TestMain
has run rather than before, allowing a test package to adjust the amount of test parallelism itself. Another idea (courtesy of @mvdan) is that the testing
package could avoid starting new parallel tests when the number of goroutines grows large.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (16 by maintainers)
Commits related to this issue
- fix: lazily init zstd reader to allow -race runs The zstd.NewReader function spawns multiple goroutines. Ref: golang/go#47056 — committed to ClickHouse/ch-go by ernado 3 years ago
I just happen to have a redesigned tsan runtime that supports infinite number of goroutines (also twice faster and consumes less memory). However, there is still a long road to upstreaming it. If you feel adventurous enough, you may try: https://go-review.googlesource.com/c/go/+/333529 or build yourself from: https://github.com/dvyukov/llvm-project/pull/4 (only linux/amd64 is supported for now)
I’ve uploaded a new version at https://go-review.googlesource.com/c/go/+/333529. A number of bugs were fixed. Testing is welcome.
The linux/amd64 version of race runtime is submitted. Besides removing the goroutine limit, it should also be ~2x-ish faster and consume less memory. It would be great to hear any success stories. Should be also much more suitable for running end-to-end systems tests and server canaries.
Change https://golang.org/cl/333529 mentions this issue:
runtime/race: update runtime (v3)