istio: Memory leak in the Istio sidecar proxy

Bug Description

We have been running Istio sidecar proxies that are consistently leaking memory over the span of a few days (as shown in the chart that uses container_memory_working_set_bytes)

We have observed this across many different services. This issue may be related to #33073

The memory profiling (inuse-space and inuse-objects) for the istio-proxy as instructed on this page looks like this -

heap_profile_inuse_space heap_profile_inuse_objects

I can provide any other details if you need them.

Version

$ istioctl version  
client version: 1.13.1  
control plane version: 1.9.8  
data plane version: 1.9.8 (340 proxies), 1.8.5 (14 proxies), 1.9.6 (27 proxies), 1.8.6 (9 proxies)

Additional Information

No response

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 31 (9 by maintainers)

Most upvoted comments

Opened a new issue: https://github.com/istio/istio/issues/48028

Thanks

owais on Nov 24, 2023

We have faced exactly the same issue and found that the issue was that we were overriding connections on our application side without properly closing it.

It seems that Istio holds those connections open forever (or until it consumes all available memory and dies).

Making sure we closed all the connections on the application side seems to have solved it.

Is it possible to set idle read/write timeouts?

rafaelsierra on Oct 24, 2023