icevision: Predict memory leak
🐛 Bug
The prediction contains a reference to the image array. If any stage of the workflow starts accumulating the predictions objects (as it currently happens in COCOMetric this will result in a memory leak.
COCOMetric currently uses a “quickfix” to make sure the images are not being stored (take a look at accumulate), but we need a more general solution.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (12 by maintainers)
a temoporary solution that seems to reduce memory leakage is to use num_workers not larger than the number of physical cores of the cpu as described here https://discuss.pytorch.org/t/num-workers-in-dataloader-will-increase-memory-usage/28522/9
might be linked with https://github.com/pytorch/pytorch/issues/13246 again, sorry Lucas 😄