server: GPU memory is not released by Triton
Description
Triton does not clear or release GPU memory when there is a pause in inference. In the attached diagrams the same model is being used. It is served via ONNX.
The GPU memory keeps growing when I do not use the TensorRT optimizations to limit the workspace memory. With the TensorRT optimization it limits the memory, but not fully. When I limit it to 7 GB, 11 GB of GPU memory are used, but that stays constant.
My model config is:
name: "model_name"
platform: "onnxruntime_onnx"
backend: "onnxruntime"
default_model_filename: "model.onnx"
version_policy: {
all: {}
}
instance_group [
{
count: 2
kind: KIND_GPU
}
]
max_batch_size: 1000
dynamic_batching { }
Triton Information 24.06-py3
Are you using the Triton container or did you build it yourself?
Container
To Reproduce
Expected behavior A clear and concise description of what you expected to happen.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 16
Facing the same issue with
24.01and24.08@susnato Yes. I found the parameter. thank you.
parameters: { key: “ENABLE_CACHE_CLEANING” value: { string_value:“true” } }
Hi, I am glad that it worked!
The way I would configure my prod environment is - (considering the max batch size is 24 for your gpu)
parameters {{ key: "arena_extend_strategy" value: {{ string_value: "1" }} }}and keep the other configs as defaultI will keep the memory shrinkage as off because separate memory allocation for each request would increase the latency of the service. And since I will send a payload of max batch size at the start of the server, it will allocate the max cuda memory possible from the start and any other request that comes after the first request will be less than the max batch and that will re-use this already allocated memory thus reducing the latency from that point. Also since we are restricting the client side code, it will never send payload more than 24 examples so it will never throw a OOM error.
Only if the new batch is less than or equal to the previous batch size.
Yes, it’s ok to use it in production environment, you can find more about the params here
I took a quick look over your config files and since you are using default settings, you are actually allocating more memory than you need (in the power of 2), so can you also add this line to the end of the config file too and restart the server?
This should drastically reduce the amount of memory you allocate for each of your infer calls. and you should be able to use batch size of 50+ (considering you had 8 as max before)
Hi, can you please add this line at the end of config.pbtxt in triton and restart the server and see if this works or not?
I might be wrong with the syntax here but basically you need to turn on the shrinkage for Cuda memory arena.
Please let me know if this works otherwise I will test it myself tomorrow and let you know the correct syntax which worked for me in past.
Same here, is there any news ?
using ONNX backend and a bge m3 onnx model. I fill up the gpu memory and never release…