kubernetes: High cpu usage in json.compact call

What happened?

I was debugging performance of case with heavy watch traffic of CRD objects and found interesting suboptimality: we burn significant amount of CPU on json.compact calls that deserialize JSONs to validate correctness of already serialized objects:

It looks like while in WatchServer.ServeHTTP we manage to reuse serialized objects (cachingObject) in object serialization: https://github.com/kubernetes/kubernetes/blob/7af5a7bfc51d0455d8b2322ae9e72ed66fb1b8f9/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go#L247-L251, we then wrap this objects in metav1.Event object and serialize it again: https://github.com/kubernetes/kubernetes/blob/7af5a7bfc51d0455d8b2322ae9e72ed66fb1b8f9/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/watch.go#L271

While that object consists of single enum and already serialized raw object it is expected that its serialization will be cheap. And it is for protobufs where we simply copy bytes: https://github.com/kubernetes/kubernetes/blob/7af5a7bfc51d0455d8b2322ae9e72ed66fb1b8f9/staging/src/k8s.io/apimachinery/pkg/runtime/generated.pb.go#L187-L192

But for json, it’s much more expensive:

• We return already serialized object if re.Raw is set (it is on watch code path): https://github.com/kubernetes/kubernetes/blob/7af5a7bfc51d0455d8b2322ae9e72ed66fb1b8f9/staging/src/k8s.io/apimachinery/pkg/runtime/extension.go#L37-L51
• but golang’s json serializer compacts that json to check its validity: https://github.com/golang/go/blob/d8762b2f4532cc2e5ec539670b88bbc469a13938/src/encoding/json/encode.go#L498

This validation happens in each watcher separately contributing to significant CPU usage.

What did you expect to happen?

We won’t deserialize/compact objects in each watch client separately.

How can we reproduce it (as minimally and precisely as possible)?

1. Open watch for objects using json encoder
2. See pprofs.

Anything else we need to know?

No response

Kubernetes version

1.24

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)