client-go: high memory usage in Reflector.ListAndWatch
I am investigating high memory usage in one of the Kubernetes controller that use client-go. In a cluster with 15000 pods it is observed that 200-300 MB memory is used by client-go constructs. I stripped down my code interaction with client-go to include just below lines with no listing of pods and no resource event handler registered.
informerFactory := informers.NewSharedInformerFactory(kr.Client, 0)
podInformer := informerFactory.Core().V1().Pods().Informer()
I still see heavy memory usage. Attached is the pprof heap profile.
api, apimachinery, client-go all at v0.19.2
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (3 by maintainers)
Also there seems to be unbounded memory growth resulting due to listing of the cached objects particularly pods over long period of time.
Attached is heap profile.
pod.informer.longrunning.pb.gz
Appriciate any insight into how to debug this.