go: runtime: "out of memory" in heapsampling.go on 386 and wasm since 2021-11-05
greplogs --dashboard -md -l -e FAIL\\s+heapsampling\\.go --since=2021-01-01
2021-11-12T18:57:22-b1b6d92/windows-386-2008 2021-11-12T16:58:34-95d0657/windows-386-2008 2021-11-05T20:59:43-dbd3cf8/windows-386-2008 2021-11-05T18:20:07-53bab19/windows-386-2008 2021-11-05T17:46:59-e48e4b4/windows-386-2008 2021-11-05T15:52:40-a0d661a/windows-386-2008
(CC @bufflig)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (18 by maintainers)
It’s not so surprising to me. 😃 I encountered this failure on js/wasm when landing the SetMemoryLimit changes. Basically, js/wasm is very slow and single-threaded, so it can trigger the GC CPU limiter if it gets particularly unlucky, so it’s especially sensitive to bugs in the GC CPU limiter, at least when a test allocates in a loop like this one.
I suspect https://go.dev/cl/404304 will help on that front, since there are still bugs. 😃
I’ve been investigating heap overrun issues in a test I wrote for #48409 that appear to be similar to #37331.
So far, I’ve discovered that any test that allocates in a loop can somewhat easily overrun the heap in an unbounded manner due to an OS scheduler preemption (even on undersubscribed systems) during mark termination. I believe I have sufficient evidence to narrow it down to an OS scheduler preemption at this point, too. I’ve been trying to obtain a scheduler trace and prove it for certain. It reproduces more readily on VMs, but those traces seem to be busted. I can reproduce on a dedicated machine as well, but not while tracing is active. (I ran it continuously over the weekend trying to obtain a trace – no dice… on Linux, anyway.)
This bug frankly looks a lot like those cases, and is consistent with previous behavior I’ve observed when investigating this bug (with smaller spikes).
It’s unfortunate that we don’t know what caused this, but at this point I think we can close this issue. We’ll reopen it if the issue returns.