onnxruntime: How to release gpu memory without exiting the process？
after run these code，how to free gpu memory manually without exiting the process？
ort_inputs = {self.ort_session.get_inputs()[0].name: images}
ort_outs = self.ort_session.run(None, ort_inputs)
System information
- Ubuntu 20.04:
- pip install onnxruntime-gpu:
- onnxruntime-gpu 1.7.0:
- Python3.8
- cuda11.0 cudnn11.2
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 15 (2 by maintainers)
I tried deleting the onnxruntime.InferenceSession using “del ort_session” but the GPU still shows memory usage. I want to clear GPU memory for other models within the same python process. @skottmckay
We use a memory pool for the GPU memory. That is freed when the ORT session is deleted.
Currently there’s no mechanism to explicitly free memory that the session is using whilst keeping the session around.