server: GPU memory is not released by Triton

Description

Triton does not clear or release GPU memory when there is a pause in inference. In the attached diagrams the same model is being used. It is served via ONNX.

image (1)

image (2)

The GPU memory keeps growing when I do not use the TensorRT optimizations to limit the workspace memory. With the TensorRT optimization it limits the memory, but not fully. When I limit it to 7 GB, 11 GB of GPU memory are used, but that stays constant.

My model config is:

name: "model_name"
platform: "onnxruntime_onnx"
backend: "onnxruntime"
default_model_filename: "model.onnx"
version_policy: {
      all: {}
}
instance_group [
  {
    count: 2
    kind: KIND_GPU
  }
]
max_batch_size: 1000

dynamic_batching { }

Triton Information 24.06-py3

Are you using the Triton container or did you build it yourself?

Container

To Reproduce

Expected behavior A clear and concise description of what you expected to happen.

About this issue

Most upvoted comments

Facing the same issue with 24.01 and 24.08

@susnato Yes. I found the parameter. thank you.

parameters: { key: “ENABLE_CACHE_CLEANING” value: { string_value:“true” } }

Hi, I am glad that it worked!

The way I would configure my prod environment is - (considering the max batch size is 24 for your gpu)

  1. Change only the memory allocation strategy - parameters {{ key: "arena_extend_strategy" value: {{ string_value: "1" }} }} and keep the other configs as default
  2. pass a payload with max batch size of 24 just right after the server is started.
  3. Then restrict at the client side so that the client will never be able to post a payload more than 24 examples as a batch

I will keep the memory shrinkage as off because separate memory allocation for each request would increase the latency of the service. And since I will send a payload of max batch size at the start of the server, it will allocate the max cuda memory possible from the start and any other request that comes after the first request will be less than the max batch and that will re-use this already allocated memory thus reducing the latency from that point. Also since we are restricting the client side code, it will never send payload more than 24 examples so it will never throw a OOM error.

If I understood correctly your explanation, without memory won’t be freed but on new batch ORT will internally decide to override past batch with it’s own optimized algo.

Only if the new batch is less than or equal to the previous batch size.

How did you find out this param ? Is there other params like this one (basic need to know) I need to know about ? Is it ok to use it in prod env ?

Yes, it’s ok to use it in production environment, you can find more about the params here

I took a quick look over your config files and since you are using default settings, you are actually allocating more memory than you need (in the power of 2), so can you also add this line to the end of the config file too and restart the server?

parameters {{ key: "arena_extend_strategy" value: {{ string_value: "1" }} }}

This should drastically reduce the amount of memory you allocate for each of your infer calls. and you should be able to use batch size of 50+ (considering you had 8 as max before)

Hi, can you please add this line at the end of config.pbtxt in triton and restart the server and see if this works or not?

parameters { key: "enable_mem_arena" value: { string_value: "1" } }
parameters { key: "memory.enable_memory_arena_shrinkage" value: { string_value: "gpu:0" } }

I might be wrong with the syntax here but basically you need to turn on the shrinkage for Cuda memory arena.

Please let me know if this works otherwise I will test it myself tomorrow and let you know the correct syntax which worked for me in past.

Same here, is there any news ?

using ONNX backend and a bge m3 onnx model. I fill up the gpu memory and never release… image