go: runtime: performance degradation for versions > 1.4.x

This issue is a follow-up to a thread that started on the mailing list here: https://groups.google.com/forum/#!topic/golang-dev/sfwfQTu7eM4

We recently reverted back to Go 1.4.3 for the most recent release of InfluxDB after testing against 1.5.x and collecting feedback from both our own benchmarks and the experiences of InfluxDB users. There were significant performance hits with both 1.5.x and 1.6beta1, across a variety of dimensions. For example, with write performance, we saw a 33% reduction in overall throughput, coupled with a 26% increase in average response time. We expected that the GC improvements would lead to a reduction in response times, but that hasn’t proven to be the case. It’s far from exhaustive, but here are the findings from a medium-scale write test:

version	points/sec	response time
Go 1.4.3	398,747	33.97ms
Go 1.5.3	300,750	43.12ms
Go 1.6rc1	315,545	39.44ms

As requested in the mailing list thread, we ran the test with GODEBUG=gctrace=1 and recorded the output alongside our log messages for context. It doesn’t look like the GC-specific lines should be too hard to grep out if our other log lines are too noisy.

We also ran with cpuprofile and memprofile on, and collected the output. We’ve included results for these tests on all three versions of Go listed above, and provided the compiled binaries as well. You can find them in this Google Drive folder: https://drive.google.com/open?id=0B3YrTka8Ai_McUNCak1YSW9ZM28

I hope this is enough information to assist in debugging the underlying issue. Definitely let me know if there are any other tests I can run to help. We’re looking forward to getting to better performance than Go 1.4.3!