go: cmd/compile: out of memory compiling cmd/compile/internal/ssa with 1GB RAM

Building tip on Ubuntu 18.04 on a Digital Ocean VM with one 1GB of RAM at commit 83dfc3b0

phil@spare0:~/go/src$ uname -a
Linux spare0 4.15.0-30-generic #32-Ubuntu SMP Thu Jul 26 17:42:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I can reproduce this out-of-memory condition 100% of the time (in the prove pass in SSA):

Building Go cmd/dist using /home/phil/go-bootstrap.
Building Go toolchain1 using /home/phil/go-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
# cmd/compile/internal/ssa
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x9f727f, 0x16)
        /home/phil/go/src/runtime/panic.go:608 +0x72
runtime.sysMap(0xc02c000000, 0x4000000, 0xe7d258)
        /home/phil/go/src/runtime/mem_linux.go:156 +0xc7
runtime.(*mheap).sysAlloc(0xe5a580, 0x4000000, 0xe5a598, 0x7f0d87065308)
        /home/phil/go/src/runtime/malloc.go:619 +0x1c7
runtime.(*mheap).grow(0xe5a580, 0x3, 0x0)
        /home/phil/go/src/runtime/mheap.go:920 +0x42
runtime.(*mheap).allocSpanLocked(0xe5a580, 0x3, 0xe7d268, 0x400)
        /home/phil/go/src/runtime/mheap.go:848 +0x337
runtime.(*mheap).alloc_m(0xe5a580, 0x3, 0x50, 0x7f0d87351fff)
        /home/phil/go/src/runtime/mheap.go:692 +0x119
runtime.(*mheap).alloc.func1()
        /home/phil/go/src/runtime/mheap.go:759 +0x4c
runtime.(*mheap).alloc(0xe5a580, 0x3, 0x7f0d87010050, 0x7f0d87065270)
        /home/phil/go/src/runtime/mheap.go:758 +0x8a
runtime.(*mcentral).grow(0xe5ccb8, 0x0)
        /home/phil/go/src/runtime/mcentral.go:232 +0x94
runtime.(*mcentral).cacheSpan(0xe5ccb8, 0x1fe)
        /home/phil/go/src/runtime/mcentral.go:106 +0x2f8
runtime.(*mcache).refill(0x7f0d8b82a000, 0xc000020050)
        /home/phil/go/src/runtime/mcache.go:122 +0x95
runtime.(*mcache).nextFree.func1()
        /home/phil/go/src/runtime/malloc.go:749 +0x32
runtime.systemstack(0x455ad9)
        /home/phil/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
        /home/phil/go/src/runtime/proc.go:1229

1GB of memory has been more than enough to build the toolchain in the past.

Barring any clever ideas about how to debug this, I’ll try to bisect and hope that there was only one commit that reliably introduced this regression.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 36 (29 by maintainers)

Commits related to this issue

cmd/compile: re-use regalloc's []valState Updates #27739: reduces package ssa's allocated space by 3.77%. maxrss is harder to measure, but using best-of-three-runs as reported by /usr/bin/time -l, I... — committed to golang/go by josharian 5 years ago
cmd/compile: remove large intermediate slice from gc.scopePCs Three loops can be converted into one. Minor reviewer-recommended refactoring. Passes toolstash-check. Updates #27739. Change-Id: Ia87a... — committed to golang/go by dr2chase 5 years ago
cmd/compile: index line number tables by source file to improve sparsity This reduces allocations and also resolves some lurking inliner/inlinee line-number match problems. However, it does add about... — committed to golang/go by dr2chase 6 years ago
cmd/compile: optimize postorder name old alloc/op new alloc/op delta Template 37.1MB ± 0% 36.8MB ± 0% -0.57% (p=0.008 n=5+5) Unicode 28.1MB ± 0% 28.1MB... — committed to golang/go by josharian 5 years ago

Most upvoted comments

FTR the openbsd/arm builder has been passing again since 6081a9f landed - thanks.

4a6f656c on May 19, 2019

I think we should have a builder that has a relatively small amount of memory. Somewhat related, in https://github.com/golang/go/issues/26867 I reported how go test net OOM’d with a few gigabytes of available memory.

We can assume that Linux is fairly common on machines with less memory (small computers, routers, VMs, etc), and that a 64-bit architecture like amd64 should stress test the memory more than a 32-bit architecture would.

We already have special builders like linux-amd64-noopt, so I propose adding a linux-amd64-small. It could start at a limit of 2GB of memory, but we could lower that to 1GB or even lower once it’s in place. I presume that we could also add extra limits to it, such as:

lowering the maximum number of open file descriptors
lowering the size of /tmp
lowering the maximum number of processes created by the user
giving the machine CPU power comparable to embedded devices (e.g. dual-core 1GHz)
lowering the size of the disk

/cc @dmitshur

mvdan on Sep 19, 2018

CL 213703 should help, if it goes in.

josharian on Jan 7, 2020

@lyda please file a new issue, and we’ll investigate there. And please include a complete log and setup information. Thanks!

josharian on May 14, 2019

https://github.com/golang/go/issues/20104 is one big fix here

josharian on Dec 13, 2018