go: build: all.bash fails to saturate 6 cores
go version devel +eed2bb7 Wed Nov 2 19:21:16 2016 +0000 linux/amd64
What did you do?
./clean.bash
time ./all.bash
What did you expect to see?
I do my development on a virtualized 6-core Xeon running Ubuntu. (runtime.NumCPU()
returns 6
.)
Per @rsc’s comments on #10571, I expected “Both the build half and the test half of all.bash should consume nearly all the CPUs for most of the time.”
What did you see instead?
all.bash seems to saturate at ~2 CPU cores for a significant fraction of the run. It consumes less than twice as much user
time than real
. Oddly, sys
usage is even higher than user
; I’m not sure what to make of that.
real 8m55.199s
user 14m13.511s
sys 8m51.402s
Here’s a chart of CPU saturation over time as measured by mpstat
. You can see that there are a couple bursts of parallelism with mostly poor saturation in between. The machine doesn’t appear to be hitting swap during the build.
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 6
- Comments: 93 (19 by maintainers)
Commits related to this issue
- cmd/compile/internal/gc: make tests run faster TestAssembly takes 20s on my machine, which is too slow for normal operation. Marking as -short has its dangers (#17472), but hopefully we'll soon have ... — committed to golang/go by josharian 8 years ago
- cmd/objdump: speed up tests Rebuild cmd/objdump once instead of twice. Speeds up standalone 'go test cmd/objdump' on my machine from ~1.4s to ~1s. Updates #17751 Change-Id: I15fd79cf18c310f892bc28a... — committed to golang/go by josharian 8 years ago
- net/http: speed up tests, use t.Parallel when it's safe Before: 8.9 seconds for go test -short After: 2.8 seconds There are still 250 tests without t.Parallel, but I got the important onces using a... — committed to golang/go by bradfitz 8 years ago
- all: sprinkle t.Parallel on some slow tests I used the slowtests.go tool as described in https://golang.org/cl/32684 on packages that stood out. go test -short std drops from ~56 to ~52 seconds. Th... — committed to golang/go by bradfitz 8 years ago
- cmd/vet: parallelize tests Was 2.3 seconds. Now 1.4 seconds. Next win would be not running a child process and refactoring main so it could be called from tests easily. But that would also require r... — committed to golang/go by bradfitz 8 years ago
- cmd/go: parallelize some tests Cuts tests from 35 to 25 seconds. Many of these could be parallel if the test runner were modified to give each test its own workdir cloned from the tempdir files they... — committed to golang/go by bradfitz 8 years ago
- all: sprinkle t.Parallel on some slow tests I used the slowtests.go tool as described in https://golang.org/cl/32684 on packages that stood out. go test -short std drops from ~56 to ~52 seconds. Th... — committed to FiloSottile/go by bradfitz 8 years ago
- all: sprinkle t.Parallel on some slow tests I used the slowtests.go tool as described in https://golang.org/cl/32684 on packages that stood out. go test -short std drops from ~56 to ~52 seconds. Th... — committed to FiloSottile/go by bradfitz 8 years ago
- cmd/go: convert tests using testdata/src/syntaxerror to scripts This includes TestMatchesNoTestsDoesNotOverrideBuildFailure and TestErrorMessageForSyntaxErrorInTestGoFileSaysFAIL. Convert the tests t... — committed to golang/go by matloob 5 years ago
- cmd/go: remove TestGoTestDetectsTestOnlyImportCycles The error that's tested in this test is also tested in list_test_err.txt which uses go list -test -deps. Because both commands are just loading pa... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestRunInternal to the script test framework This conversion is a bit weird, because the original test runs in the cmd/go directory, while the script test runs in the GOPATH directory... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestGoBuildNotMain to script framework Part of converting all tests to script framework to improve test parallelism. Updates #36320 Updates #17751 Change-Id: Icd62dc8db55bec52ad326b... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestGoTestImportErrorStack to the script framework The literal string match had to be turned into a regexp, otherwise pretty straightforward. Part of converting all tests to script f... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestIssue7108 to the script framework Part of converting all tests to script framework to improve test parallelism. Updates #36320 Updates #17751 Change-Id: Ic16a0c6c1eabeba003b9cff... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestRunPkg to the script framework Part of the effort to convert all non-parallel cmd/go tests to the script framework. Updates #17751 Updates #36320 Change-Id: I2bc0b1e5c03e2c49b5c... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestGoGenerateBadImports to script framework Part of converting all tests to script framework to improve test parallelism. Updates #36320 Updates #17751 Change-Id: Id1afafad2063d917... — committed to golang/go by matloob 5 years ago
- cmd/go: parallellize more test cases Sprinkle in some more calls to tg.Parallel in the cmd/go tests. These are the easy cases that don't operate in the testdata directory. I think the best thing to ... — committed to golang/go by matloob 5 years ago
- cmd/go: convert some testcases to the script framework This change converts TestFatalInBenchmarkCauseNonZeroExitStatus, TestBenchmarkLabels and TestWildCardsDoNotLookInUselessDirectories to the scrip... — committed to golang/go by matloob 5 years ago
- cmd/go: convert TestGoTestWithoutTests to the script framework Part of converting all tests to script framework to improve test parallelism. Updates #36320 Updates #17751 Change-Id: Ibc3cb3823bd1c1... — committed to golang/go by matloob 4 years ago
- cmd/go: remove tests using testdata/flag_test.go Part of converting all tests to script framework to improve test parallelism. Updates #36320 Updates #17751 Change-Id: I6573185cf14f298c51f76265f18a... — committed to golang/go by matloob 4 years ago
The initial build transient seems to saturate a bit better, and a few of the tests spike to saturation, but the overall utilization is still pretty similar.
From a clean build at tip: