go: testing: benchmark iteration reports incorrectly
What version of Go are you using (go version
)?
$ go version go version devel +8266570ba7 Fri Sep 25 19:10:05 2020 +0000 darwin/amd64
Does this issue reproduce with the latest release?
Yes. 1.15.2.
What operating system and processor architecture are you using (go env
)?
go env
Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/Users/changkun/Library/Caches/go-build" GOENV="/Users/changkun/Library/Application Support/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GOINSECURE="" GOMODCACHE="/Users/changkun/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/changkun/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/Users/changkun/dev/godev/go-gerrit" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/Users/changkun/dev/godev/go-gerrit/pkg/tool/darwin_amd64" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/Users/changkun/dev/godev/go-gerrit/src/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/jd/vym7tt9s2_379d4ccjcj1xrw0000gn/T/go-build637901099=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do?
According to the doc:
The benchmark function must run the target code b.N times. During benchmark execution, b.N is adjusted until the benchmark function lasts long enough to be timed reliably. The output
BenchmarkRandInt-8 68453040 17.8 ns/op
means that the loop ran 68453040 times at a speed of 17.8 ns per loop.
But the following test fails and the benchmark runs the target code more than b.N times.
// main_test.go
func TestResultIteration(t *testing.T) {
var v int32
r := testing.Benchmark(func(b *testing.B) {
for i := 0; i < b.N; i++ {
atomic.AddInt32(&v, 1)
}
})
if r.N != int(v) {
t.Fatalf("incorrect iteration count, want %d got %d", v, r.N)
}
}
What did you expect to see?
PASS
What did you see instead?
=== RUN TestResultIteration
main_test.go:27: incorrect iteration count, want 359815341 got 258805240
--- FAIL: TestResultIteration (1.67s)
FAIL
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (19 by maintainers)
Running the benchmark incrementally would potentially be faster, but not necessarily more consistent. Many benchmarks have a non-trivial transient at the start of the benchmark (for example, due to CPU cache misses). Summing the results of multiple incremental runs would also sum the transient effects, whereas taking only the last run (with the final computed
N
) includes only a single transient, and should therefore have less noise for certain benchmarks.I don’t think this is a bug,
testing.Benchmark
returns aBenchmarkResult
which is the result of a benchmark run.https://golang.org/pkg/testing/#BenchmarkResult