rf-detr: Bug: Zero AP/AR scores during training & crash due to missing 'checkpoint_best_regular.pth'
Search before asking
- I have searched the RF-DETR issues and found no similar bug report.
Bug
When training the RFDETRBase model on a custom dataset in COCO format, the evaluation step using pycocotools consistently reports zero for all Average Precision (AP) and Average Recall (AR) metrics during every evaluation cycle.
Subsequently, at the very end of the training process, the script fails with a FileNotFoundError because it cannot find the checkpoint_best_regular.pth file to copy. This suggests that due to the zero scores, no checkpoint was ever considered the “best” to be saved under that name.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
The code failded with the following error :
Traceback (most recent call last):
File "/data/rfdetr/train.py", line 26, in <module>
File "/data/rfdetr/venv/lib/python3.10/site-packages/rfdetr/detr.py", line 36, in train
self.train_from_config(config, **kwargs)
File "/data/rfdetr/venv/lib/python3.10/site-packages/rfdetr/detr.py", line 74, in train_from_config
self.model.train(
File "/data/rfdetr/venv/lib/python3.10/site-packages/rfdetr/main.py", line 405, in train
shutil.copy2(output_dir / 'checkpoint_best_regular.pth', output_dir / 'checkpoint_best_total.pth')
File "/home/user/anaconda3/lib/python3.10/shutil.py", line 434, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/user/anaconda3/lib/python3.10/shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'output/checkpoint_best_regular.pth'
Environment
- rfdetr : 1.0.8
- os: ubuntu 22.04
- python: 3.10.9
- torch: 2.6.0
- GPU: RTX 6000 Ada
Minimal Reproducible Example
from rfdetr import RFDETRBase
model = RFDETRBase()
model.train(
dataset_dir='./custom_dataset',
epochs=5,
batch_size=16,
grad_accum_steps=1,
lr=1e-4,
output_dir='./output',
device='cuda:1'
)
The custom dataset structure is as follow:
custom_dataset/
├── test
├── train
└── valid
And in each subset folder, there is the images and _annotations.coco.json
Additional
No response
Are you willing to submit a PR?
- Yes, I’d like to help by submitting a PR!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 18
Yep, thanks for your quick response! 🚀
Still the same issue despite modifying my annotations as follows and adapting each bounding box annotation accordingly. I’m continuing to investigate and will keep you updated. Let me know if you have any other ideas.
Here is the terminal logs: rfdetr_logs.txt And the logs from the training: train_logs.txt