zed-sdk: Memory Leak In SVO Reading

Hi @AlexanderKhazatsky,

We found the cause of the issue. It isn’t actually a memory leak, but this is due to how glibc manages memory under the hood.

When malloc (or friends, calloc, realloc etc… and the C++ new operator) is used, it usually allocates memory from the process’ heap and returns a pointer to this area of memory. If the heap is not big enough or if there’s no free contiguous memory chunk of the requested size available, malloc makes a system call to brk/sbrk to ask the kernel to increase the size of the heap, and then allocates the requested memory and returns a pointer to it.

When using free (or delete) however, the memory area is marked as available again in the heap for the current process, but this generally does not return the memory back to the OS. The chunk of memory is kept for the process for future allocations. Some memory is released if the amount of free memory at the end of the heap exceeds a predefined threshold M_TRIM_THRESHOLD (which is 128kB by default). Free chunks of memory “sandwiched” between two other allocated chunks cannot be released back to the OS. When this happens, this is called heap fragmentation.

Heap fragmentation is when the heap contains multiple free memory chunks (which cannot be returned to the OS) between some other allocated chunks: X…XXX…X…X (the allocated memory chunks are represented as X and the free ones as …) Those free memory chunks can (and will, if they’re big enough) be used for future allocations in the process, but cannot be released to the OS, until all the following memory chunks in the heap are freed. For example, here, if the last X is freed, and …X (the 3 dots plus the X) is more than 128kB, this memory chunk will be released to the OS because free will automatically call brk/sbrk to reduce the heap size appropriately.

Now, actually, in glibc, when malloc is requested an amount of memory bigger than M_MMAP_THRESHOLD (128kB by default), and if a contiguous memory chuck of the requested size is not available in the heap, it uses the mmap syscall to get a memory mapping of the requested size from the OS, instead of using brk/sbrk to increase the size of the heap. In this case, the memory in released back to the OS directly when it is freed.

So, to come back to the original issue, I see 2 possibilities to force the process to give back unused memory to the system:

• malloc actually allows to tweak its behaviour via environment variables. See mallopt(3) for all the details, but in a nutshell:

  • Setting the environment variable MALLOC_TRIM_THRESHOLD_ allows to change the M_TRIM_THRESHOLD value.
  • Setting the environment variable MALLOC_MMAP_THRESHOLD_ allows to change the M_MMAP_THRESHOLD. value.

  Now you have to understand that setting those values is a tradeoff between memory usage and speed: using a low value for M_TRIM_THRESHOLD will reduce memory usage because it will give back memory to the OS more frequently, but it will increase the number of system calls. Setting M_TRIM_THRESHOLD to 0 will force the process to always give back the unused memory at the end of the heap to the OS.

  The manual says:

    Modifying M_TRIM_THRESHOLD is a trade-off between increasing the number of system calls (when the parameter is set low) and wasting unused memory at the top of the heap (when the parameter is set high).
  

  You can try different values and see what works best for you. But in your case, you want to try values lower than 128*1024, which is the default.

• Use an alternative implementation of malloc and friends, with improved memory management, for example tcmalloc. I tried it on the SDK and it seems to work pretty well. It uses less memory and is faster. It’s not planned to integrate it into the SDK but maybe we will discuss it if it shows to have real benefits. Don’t hesitate to give some feedback about it if you use it with the SDK. To use it with your current program, install it first (sudo apt install libtcmalloc-minimal4), and use the LD_PRELOAD environment variable to run your program with it:

    LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4 ./your_program
  

  It will override the implementations of malloc, calloc, new etc. See https://goog-perftools.sourceforge.net/doc/tcmalloc.html for more infos and docs.

The fact that the behaviour is different between Ubuntu 22.04 and Ubuntu 20.04 is probably due to an improvement in the memory management in glibc (libc6 2.31 on ubuntu 20.04 vs libc6 2.35 on ubuntu 22.04).

If you want to learn more about this, here are some resources I found helpful to understand everything:

• http://www.cplusplus-soup.com/2010/01/freedelete-not-returning-memory-to-os.html
• https://www.reddit.com/r/coding/comments/2ditbr/comment/cjpwfff/?utm_source=share&utm_medium=web2x&context=3
• https://www.gnu.org/software/libc/manual/html_node/Freeing-after-Malloc.html#Freeing-after-Malloc
• https://stackoverflow.com/questions/2215259/will-malloc-implementations-return-free-ed-memory-back-to-the-system
• https://stackoverflow.com/questions/48358229/how-can-i-get-a-guarantee-that-when-a-memory-is-freed-the-os-will-reclaim-that
• https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use/

Just to clarify, I’m using the Python API

In the zip file above, I included readings from a zed mini (which doesn’t leak) as well as two zed 2’s which do memory leak.

@AlexanderKhazatsky Thanks for the additional information, we’ll investigate on our side to try to replicate this. In the meantime, if you find other elements don’t hesitate to share them. Would it be possible to share one of your SVO file that triggers this issue? You can send it privately to support@stereolabs.com if needed.