istio: Memory leak in istio-proxy

cpu.zip

(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description Istio-proxy is consuming increasing amount of memory for an application with high HTTP traffic (7000 qps). Cpu utilization remains at 100% after traffic ends and memory increases continuously. Connection remain hanging in Close_wait state.

[ ] Docs [ ] Installation [X] Networking [X] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior Istio-proxy’s memory shouldn’t increase continuously. Cpu usage should decrease after traffic is ended and connections should close

Steps to reproduce the bug I have uploaded the cpu profile taken from the envoy with the profiler tool. Also the following logs appear in the sidecar logs repeatedly 2021-01-29T10:18:18.203791Z trace envoy http2 [C1039] sent frame type=3 2021-01-29T10:18:18.203794Z debug envoy http2 [C1039] sent reset code=0 2021-01-29T10:18:18.203797Z trace envoy http2 [C1039] about to send frame type=3, flags=0 2021-01-29T10:18:18.203800Z trace envoy http2 [C1039] send data: bytes=13 2021-01-29T10:18:18.203803Z trace envoy connection [C1039] writing 13 bytes, end_stream false 2021-01-29T10:18:18.203791Z trace envoy http2 [C1039] sent frame type=3 2021-01-29T10:18:18.203794Z debug envoy http2 [C1039] sent reset code=0 2021-01-29T10:18:18.203797Z trace envoy http2 [C1039] about to send frame type=3, flags=0 2021-01-29T10:18:18.203800Z trace envoy http2 [C1039] send data: bytes=13 2021-01-29T10:18:18.203803Z trace envoy connection [C1039] writing 13 bytes, end_stream false

Version control plane version: 1.7.4 data plane version: 1.7.4 (3 proxies)

Client Version: v1.18.9 Server Version: v1.18.9

v3.3.3+g5972293

How was Istio installed? helm install

Environment where the bug was observed (cloud vendor, OS, etc)

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.