istio: Protocol sniffing causes excessive memory usage/pilot CPU usage
When protocol sniffing is enabled, Envoy will use very high memory usage.
Some numbers:
- With 20 services, memory usage went from 40mb->80mb
- With 300 services, memory usage went from 60mb to 800mb
- With mixer disabled and 300 services, memory was around 400mb
These numbers are all taken from an Envoy receiving no traffic. Envoy will have high allocated as well, not just high heap usage (as seen in previous cases where it would use a lot of memory at startup then free it).
When switching from http to auto, we will no longer use a shared listener/HCM. So previously, if we had 100 HTTP services on port 80, we had 1 listener and 100 routes. Now, we will have 100 listeners, with 100 HCMs, and 100 routes.
Envoy may internally also store a copy of the routes in the HCM as well, so that would result in 100*100 route configs (this is a theory only)
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 32 (31 by maintainers)
Well Yan is working on that now, so hopefully it’s fairly simple and we can have something soon. We are building the release tomorrow but not planning to ship until Tuesday afaik. If we could fix by tomorrow or Monday then I suspect we could convince the release managers to delay the release by a day or 2, but I don’t think they will accept indefinitely delaying the release until this is fixed.
On Thu, Sep 5, 2019, 8:25 PM Shriram Rajagopalan notifications@github.com wrote: