go: testing: provide a way to work around "race: limit on 8128 simultaneously alive goroutines is exceeded" error

$ go version
go version devel go1.17-dc00dc6c6b Thu Jun 10 12:41:37 2021 +0000 linux/amd64

Issue #38184 seems to be a reasonably hard limit in the race detector library, but it’s not always easy to work around it. In tests, particularly, this issue more commonly arises in higher level tests that run more slowly and hence use testing.T.Parallel to speed themselves up. But this means that if a large instance is used to run tests (for example in continuous integration), we might see more simultaneously alive goroutines because the value of GOMAXPROCS is greater so more tests will be run in parallel, leading to this issue arising.

There are a few current possibilities for working around this at present, although none of them feel great:

• avoid calling testing.T.Parallel when the race detector is enabled, but this disables parallelism completely, which feels like overkill.
• run some packages with a different value for the -test.parallel flag, but this means that the top level CI script needs to be changed (you’ll no longer be able to run go test -race ./...) which has maintainability issues.
• update the value of -test.parallel within a given package by using flag.Lookup and setting the flag directly, but this is a nasty hack.

Ideally, we’d be able to configure the maximum number of active goroutines allowed, but failing that, another possibility might be to make the default value of testing.parallel come from runtime.GOMAXPROCS after TestMain has run rather than before, allowing a test package to adjust the amount of test parallelism itself. Another idea (courtesy of @mvdan) is that the testing package could avoid starting new parallel tests when the number of goroutines grows large.