server: Python backend SHM memory leak
Description I am encountering two possibly related issues with the Python backend and shared memory:
- During operation, the shared memory usage keeps growing, leading to errors. It looks like the shared memory regions allocated by the Python backend for its inputs are not recycled. I understand the SHM region grows based on the size of the inputs, but this is an issue especially when multiple model instances are running. Also, it is possible the region grows beyond the largest input if memory is leaked instead of re-used.
- After the Triton container is terminated, allocated shared memory regions remain in
/dev/shm
Triton Information What version of Triton are you using? 2.47.0
Are you using the Triton container or did you build it yourself? Official containers:
- 24.06-py3
- 24.06-trtllm-python-py3
To Reproduce I encountered the issue with any Python-based model I tried:
- Python models for image pre-processing
- Python BLS models
- TensorRT-LLM models using the Python backend (I don’t think I can use the tensortt_llm backend for my use case - multimodal model)
Expected behavior
- Shm regions would be shrunk, or at least wouldn’t grow indefinitely (arena-style allocator?)
- Shm regions would be de-allocated when the model shuts down
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 24 (9 by maintainers)
@gerasim13, thank you. I can reproduce the issue.
I hope this helps: https://github.com/gerasim13/triton-server-shm-bug-investigation
We are facing the same issues in our models. Any more updates on this?
Also for the second issue where
/dev/shmwill not be cleaned after container restarts. If you are in k8s environment, we’ve used a hacky way to clean it once the container restarted so at least container won’t end up CLBO because it has no memory available