mmdetection: CUDA out of memory

Dear All ,

I have a question reagrd the problem of CUDA out of the memory.

I try to train a CentriapetalNet on my custom dataset , which has the same format of COCO dataset. this is the config file : https://github.com/open-mmlab/mmdetection/tree/master/configs/centripetalnet I got such error massage on CUDA memory , I changed the following code from 63 to 164

data = dict(
    samples_per_gpu=6,
    workers_per_gpu=3,

I noticed that changing such batch size help in reduce the CUDA memory needs during the training. but it is still not working , I did not understand the theory behind . also I really confused what should I do to continues the training.

I read this document : https://mmdetection.readthedocs.io/en/latest/faq.html , the part of CUDA memory

and i did not know how to change those three items described there.

any help will be highly appreciated.

cuda out of memory

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 30 (8 by maintainers)

Most upvoted comments

scikit-image and pillow has resize functions, or if you have a small dataset, you can resize the images manually (by photo editor software) to make it work at first.