kubernetes: etcd3 watcher doesn't scale

I did some scalability tests with etcd3 enabled and it seems that watcher.go pretty much doesn’t scale.

I think the problem we have is with tranform function: https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/etcd3/watcher.go#L234

So to give you some numbers, in 2000-node kubemark, I started cluster at:

I0928 09:48:36.043709    3547 config.go:404] Will report 10.240.0.24 as public IP address.
I0928 09:48:36.045790    3547 server.go:328] Initalizing cache sizes based on 120000MB limit

and 12 second later I started getting:

W0928 09:48:48.398584    3547 watcher.go:319] Fast watcher, slow processing. Number of buffered events: 100.Probably caused by slow decoding, user not receiving fast, or other processing logic

I have millions of such lines in my logs.

Note that this was empty cluster - there weren’t any pods at that time in the system - so it was mostly traffic from nodes.

This is pretty much a blocker for launching etcd3 - we can’t launch with worse performance than etcd2

@kubernetes/sig-scalability @xiang90 @hongchaodeng @timothysc