istio: Istio uses significantly high memory
Bug Description
We upgraded to the recent Istio version and immediately we saw a jump of Istiod memory usage. We have increased the memory to 14G and still it is not sufficient. Ultimately it gets OOM Killed.
Version
Master
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 78 (75 by maintainers)
We need to backport the fix to 1.13 and also 1.14
Hmm i’m not sure I understand the CPU metric @ramaraochavali is this cumulative cpu seconds? That would not be very comparable between two deployments. I would rather have average and probably max cpu utilization of istiod pods.
What I’m meaning to say is, it would be great to compare how the CPU usage is when enabling RDS_CACHE in the latest versions. Since we’re basically using caches to save computing power, i would expect memory usage to be higher, but also cpu usage to be considerably lower, if the second doesn’t hold, then caching is not working for your use case.
What I also don’t understand is when the rollout of the new Istio version is done.
Is it too much to ask to provide two separate graphs for each metric? one for memory usage of the previous version and one for the latest, one for avg cpu usage of previous version and one for the latest. It would also probably be fair to compare on similar situations: aproximate load, connected sidecars, etc.
I think I have found another possible leak in XdsCache, gonna make a PR for fixing it and try to add some tests to validate that we don’t leak these again.
@sschepens So we have deployed the fix and enabled RDS cache back. We still see Istiod getting OOM Killed. Here are the latest pprof outputs pprof.pilot-discovery.alloc_objects.alloc_space.inuse_objects.inuse_space.005.pb.gz pprof.pilot-discovery.alloc_objects.alloc_space.inuse_objects.inuse_space.006.pb.gz
Wow so if I read this right, 4gb of keys. key is 32bytes. That is 125 000 000 keys. But XDS cache size is 60000 max. So how can we have so much memory to store the key…?