go: Building go crashes on 144 core/1+TB RAM CentOS7 machine

I happen to have access to a large system and am using it to build a variety of packages. My go builds almost always fail. Details below.

What version of Go are you using (go version)?

Bootstrapping go: from here: https://storage.googleapis.com/golang/go1.4-bootstrap-20161024.tar.gz

Go from here: https://storage.googleapis.com/golang/go1.7.4.src.tar.gz

What operating system and processor architecture are you using (go env)?

[hartzelg@rpbuchop001 go]$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/isilon/Analysis/scratch/hartzelg/linuxbrew/Cellar/go/1.7.4/libexec"
GOTOOLDIR="/isilon/Analysis/scratch/hartzelg/linuxbrew/Cellar/go/1.7.4/libexec/pkg/tool/linux_amd64"
CC="/isilon/Analysis/scratch/hartzelg/linuxbrew/bin/gcc-5"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build188975734=/tmp/go-build -gno-record-gcc-switches"
CXX="/isilon/Analysis/scratch/hartzelg/linuxbrew/bin/g++-5"
CGO_ENABLED="1"

What did you do?

Building the go recipe in spack like so:

spack install go@1.7.4

(which builds and uses a bootstrapping system from the 1.4 snapshot).

It boils down to '/usr/bin/bash' 'all.bash'.

I’m seeing this in a job running via jenkins that simplifies to this:

git clone https://github.com/llnl/spack
cd spack
export PATH=`pwd`/bin:$PATH
ulimit -u 8192
spack install go@1.7.4

What did you expect to see?

A successful build.

What did you see instead?

A panic during testing. Full output (1500+ lines) in this gist.

Here’s the first few lines.

ok  	regexp	0.455s
ok  	regexp/syntax	1.000s
panic: test timed out after 3m0s

goroutine 22864 [running]:
panic(0x5be600, 0xc42017a010)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/panic.go:500 +0x1a1 fp=0xc42010cf48 sp=0xc42010ceb8
testing.startAlarm.func1()
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/testing/testing.go:918 +0x10b fp=0xc42010cfb0 sp=0xc42010cf48
runtime.goexit()
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42010cfb8 sp=0xc42010cfb0
created by time.goFunc
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/time/sleep.go:154 +0x44

goroutine 1 [chan receive, 2 minutes, locked to thread]:
runtime.gopark(0x629c68, 0xc420016598, 0x60cadd, 0xc, 0xc420110b17, 0x3)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/proc.go:259 +0x13a fp=0xc420110b10 sp=0xc420110ae0
runtime.goparkunlock(0xc420016598, 0x60cadd, 0xc, 0xc420000117, 0x3)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/proc.go:265 +0x5e fp=0xc420110b50 sp=0xc420110b10
runtime.chanrecv(0x5bc980, 0xc420016540, 0x0, 0xc420110c01, 0x47434c)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/chan.go:496 +0x2df fp=0xc420110bd8 sp=0xc420110b50
runtime.chanrecv1(0x5bc980, 0xc420016540, 0x0)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/chan.go:378 +0x35 fp=0xc420110c10 sp=0xc420110bd8
testing.(*T).Run(0xc42001a180, 0x611a7e, 0x18, 0x62a6c8, 0xc42011fc01)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/testing/testing.go:647 +0x316 fp=0xc420110cb8 sp=0xc420110c10
testing.RunTests.func1(0xc42001a180)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/testing/testing.go:793 +0x6d fp=0xc420110d08 sp=0xc420110cb8
testing.tRunner(0xc42001a180, 0xc420110de8)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/testing/testing.go:610 +0x81 fp=0xc420110d30 sp=0xc420110d08
testing.RunTests(0x6295c0, 0x70f7c0, 0xc4, 0xc4, 0x43bf04)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/testing/testing.go:799 +0x2f5 fp=0xc420110e18 sp=0xc420110d30
testing.(*M).Run(0xc420110ef8, 0x40f769)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/testing/testing.go:743 +0x85 fp=0xc420110ea0 sp=0xc420110e18
runtime_test.TestMain(0xc420110ef8)
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/crash_test.go:27 +0x2f fp=0xc420110ef0 sp=0xc420110ea0
main.main()
	runtime/_test/_testmain.go:836 +0xc6 fp=0xc420110f58 sp=0xc420110ef0
runtime.main()
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/proc.go:183 +0x1f4 fp=0xc420110fb0 sp=0xc420110f58
runtime.goexit()
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420110fb8 sp=0xc420110fb0

goroutine 17 [syscall, 2 minutes, locked to thread]:
runtime.goexit()
	/tmp/hartzelg/spack-stage/spack-stage-ewK2oy/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42010efb8 sp=0xc42010efb0

Some time later (15 minutes after the test failed, I still have a go-build-related process running:

hartzelg  94605  400  0.0   4060  1740 ?        Rl   14:33  86:51 /tmp/go-build307875912/a.exe

and jenkins thinks that the job is still running.