go: runtime: MADV_HUGEPAGE causes stalls when allocating memory

/proc/<pid>/smaps_rollup can do the totalling for you.

👋 We noticed after updating to Go 1.21.0 that some of our apps were using more off heap memory than others. There was a wide gap in between the reported container memory (kubernetes) and the process RSS or go mstats heap_sys value.

We figured that this issue might be related and indeed updating to Go 1.21.1 solves the issue for us but we aren’t sure how the problem described here could contribute to larger amounts of retained off heap memory.

First, does the problem described here sound like it could also cause the issue that we are/were seeing? Second, do you have any suggestions for telemetry that we could look at to confirm? None of the usual suspects for us were showing anything apart from the aforementioned gap between the container memory and go heap memory

The graph below illustrates the gap, with go 1.21.0 running and then the same service being deployed with 1.20.7

And I forgot to say, thank you for your time and effort in looking into this @dominikh!

Go 1.21.1 does indeed solve the issue for us.

Glad to hear!

Are there any metrics that we could have helped us track this down further, or similar issues in the future? All we were able to tell was that the memory was being retained somewhere “off heap” but we had little visibility into what it was.

Unfortunately Linux doesn’t provide a very good way to observe how much memory is going to huge pages. You can occasionally dump /proc/<pid>/smaps and total up the AnonHugePages count. Other than that, I don’t think there’s a lot you can do. 😦

Fortunately, I don’t think you’ll have to worry about this being a problem from Go in the future. Now that we have a better understanding of the landscape of hugepage-related madvise syscalls, I don’t think we’ll be trying to explicitly back the heap with hugepages outside of specific cases and only a best-effort basis (like Go 1.21.1 now does).

Go 1.21.1 does indeed solve the issue for us.

another way to check if you’re affected, check the output of:

Below is the output:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

Are there any metrics that we could have helped us track this down further, or similar issues in the future? All we were able to tell was that the memory was being retained somewhere “off heap” but we had little visibility into what it was.

Meanwhile I can reliably reproduce the issue with Go at master, and not at all with Go 1.20. In fact, seeing such high latencies with Go 1.20 seems particularly weird, as it’s not actually making use of huge pages much at all.

The problem is very sensitive to the amount of memory fragmentation, which can make it difficult to trigger. There have to be few enough (or none) allocations available for huge pages, and “direct reclaim” must not be able to quickly merge pages, either.

I did end up having more luck with Michael’s stress test than with stress. At this point, running 10 instances of Michael’s stress test — compiled with Go master — followed by my reproducer, leads to a maximum pause of 1.29s when the reproducer is built with Go master, and ~20ms when built with Go 1.20.6, with the 20ms pauses likely being due to Linux scheduler pressure, as the stress test keeps all cores busy.

It’s also worth noting that the stress test acts as an easier way of reproducing the problem, not a requirement. I originally reproduced the problem just by having a lot of typical, heavy desktop software open (Firefox with a significant amount of tabs, Discord, Slack), which left memory quite fragmented. The stress test is a much easier way of using up large allocations.