server: Python backend SHM memory leak

Description I am encountering two possibly related issues with the Python backend and shared memory:

  1. During operation, the shared memory usage keeps growing, leading to errors. It looks like the shared memory regions allocated by the Python backend for its inputs are not recycled. I understand the SHM region grows based on the size of the inputs, but this is an issue especially when multiple model instances are running. Also, it is possible the region grows beyond the largest input if memory is leaked instead of re-used.
  2. After the Triton container is terminated, allocated shared memory regions remain in /dev/shm

Triton Information What version of Triton are you using? 2.47.0

Are you using the Triton container or did you build it yourself? Official containers:

  • 24.06-py3
  • 24.06-trtllm-python-py3

To Reproduce I encountered the issue with any Python-based model I tried:

  • Python models for image pre-processing
  • Python BLS models
  • TensorRT-LLM models using the Python backend (I don’t think I can use the tensortt_llm backend for my use case - multimodal model)

Expected behavior

  1. Shm regions would be shrunk, or at least wouldn’t grow indefinitely (arena-style allocator?)
  2. Shm regions would be de-allocated when the model shuts down

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 4
  • Comments: 24 (9 by maintainers)

Most upvoted comments

@gerasim13, thank you. I can reproduce the issue.

@gerasim13, if possible could you please help us with the issue repro to try from our end? Thank you.

I hope this helps: https://github.com/gerasim13/triton-server-shm-bug-investigation

We are facing the same issues in our models. Any more updates on this?

Also for the second issue where /dev/shm will not be cleaned after container restarts. If you are in k8s environment, we’ve used a hacky way to clean it once the container restarted so at least container won’t end up CLBO because it has no memory available

                  "lifecycle": {
                     "postStart": {
                        "exec": {
                           "command": ["/bin/sh", "-c", "rm -f /dev/shm/*"]
                        }
                     }
                  },