facenet-pytorch: memory leak
Hello, Im facing a memory leak and I can’t find out why. I am simply looping through a lot of images and it gradually fills all my memory, this is my setup: Version facenet-pytorch==2.0.1
mtcnn = MTCNN(image_size=64, keep_all=True)
resnet = InceptionResnetV1(pretrained='vggface2').eval()
for nth, img_path in enumerate(img_paths):
img = Image.open(img_path.resolve())
boxes, probs = mtcnn.detect(img)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 9
- Comments: 18 (4 by maintainers)
@marisancans @ShadowElement I’ve been able to reproduce this issue now - it doesn’t seem to happen on every system and I am not 100% sure what is happening. My guess is that it has to do with slicing a numpy array without creating a copy.
I’m in the process of tracking down the issue - if I find it and can fix, I’ll let you know.
Hi did anyone manage to find the source of this problem cause I am currently suffering from it too, I am running on cpu and the weird thing is that the memory leak doesn’t occurs all the time.
I created a pull request to fix GPU out of memory issue.
https://github.com/timesler/facenet-pytorch/pull/105
The root cause is that batch size for rnet and onet is dynamic. Sometimes the batch size for rnet input data is very large ( > 20,000). The solution is to use fixed bounded batch size.
@timesler: I have the same issue above, after digging into the codes it seems memory leak comes from PNet: https://github.com/timesler/facenet-pytorch/blob/master/models/utils/detect_face.py#L50, when I disabled it the memory leak issue disappear. Need others to take a look at this.
add
torch.cuda.empty_cache()after line 351 in mtcnn.pyI didn’t use the InceptionResnetV1 for my testing, only use the detect function of MTCNN, here is my code:
from facenet_pytorch.models.mtcnn import MTCNN from PIL import Image img = Image.open(img_path) for i in range(1000): boxes, probs = mtcnn.detect(img)
@TerryTran: There seems to be a problem in this module https://github.com/timesler/facenet-pytorch/blob/master/models/inception_resnet_v1.py As I understand it, detect_face.py is needed to find the face in the photo. I do this: model = InceptionResnetV1 (pretrained = ‘vggface2’).Eval().To(device). And the code I transfer data to get embeddings (model(data)), I get an error about the memory. Checked GPU consumption with GPUtil.showUtilization. There is a sharp increase in GPU consumption.