go: runtime: Go 1.14.rc1 3-5% performance regression from 1.13 during protobuf marshalling

$ go1.14rc1 version
go version go1.14rc1 linux/amd64
$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

This is an issue with the latest release

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/howardjohn/.cache/go-build"
GOENV="/usr/local/google/home/howardjohn/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/howardjohn/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/local/google/home/howardjohn/go/src/istio.io/istio/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build462288476=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran a benchmark for our application on go 1.13 and go 1.14rc1. performance took about a ~5% hit.

$ go test ./pilot/pkg/proxy/envoy/v2/ -bench=BenchmarkEDS -run=^$ -benchmem -benchtime=20s
goos: linux
goarch: amd64
pkg: istio.io/istio/pilot/pkg/proxy/envoy/v2
BenchmarkEDS/100/1-6              264200             90238 ns/op            8964 B/op        361 allocs/op
BenchmarkEDS/1000/1-6              30642            775560 ns/op           62452 B/op       3061 allocs/op
BenchmarkEDS/10000/1-6              3147           6909406 ns/op          604528 B/op      30086 allocs/op
BenchmarkEDS/100/100-6              3217           7292090 ns/op          876827 B/op      35645 allocs/op
BenchmarkEDS/1000/100-6              355          67245095 ns/op         6318974 B/op     307880 allocs/op
BenchmarkEDS/10000/100-6              34         657385493 ns/op        70324914 B/op    3240990 allocs/op
PASS
ok      istio.io/istio/pilot/pkg/proxy/envoy/v2 176.394s

$ go1.14rc1 test ./pilot/pkg/proxy/envoy/v2/ -bench=BenchmarkEDS -run=^$ -benchmem -benchtime=20s
goos: linux
goarch: amd64
pkg: istio.io/istio/pilot/pkg/proxy/envoy/v2
BenchmarkEDS/100/1-6              246732             97406 ns/op            8964 B/op        361 allocs/op
BenchmarkEDS/1000/1-6              29126            823575 ns/op           62454 B/op       3061 allocs/op
BenchmarkEDS/10000/1-6              3036           7492450 ns/op          604574 B/op      30087 allocs/op
BenchmarkEDS/100/100-6              3120           7801905 ns/op          876848 B/op      35645 allocs/op
BenchmarkEDS/1000/100-6              339          70948842 ns/op         6323512 B/op     307986 allocs/op
BenchmarkEDS/10000/100-6              33         689077376 ns/op        70629433 B/op    3248162 allocs/op
PASS
ok      istio.io/istio/pilot/pkg/proxy/envoy/v2 180.654

What did you expect to see?

Hopefully performance improves or remains constant

What did you see instead?

Performance decrease

Additional Info

This benchmark is basically just doing a bunch of protobuf marshalling.

Profile shows 1.14 spends 16% of time on gcBgMarkWorker, but 1.13 only 14.5%.

CPU profiles: profiles.tar.gz

Let me know if there is more info I can provide, I realize these reports typically need a lot of info to track down the root cause. You might/should be able to reproduce by running the same command after checking out the repo at this PR: https://github.com/istio/istio/pull/20899