rf-detr: Bug: Zero AP/AR scores during training & crash due to missing 'checkpoint_best_regular.pth'

Search before asking

  • I have searched the RF-DETR issues and found no similar bug report.

Bug

When training the RFDETRBase model on a custom dataset in COCO format, the evaluation step using pycocotools consistently reports zero for all Average Precision (AP) and Average Recall (AR) metrics during every evaluation cycle.

Subsequently, at the very end of the training process, the script fails with a FileNotFoundError because it cannot find the checkpoint_best_regular.pth file to copy. This suggests that due to the zero scores, no checkpoint was ever considered the “best” to be saved under that name.

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

The code failded with the following error :

Traceback (most recent call last):
  File "/data/rfdetr/train.py", line 26, in <module>
  File "/data/rfdetr/venv/lib/python3.10/site-packages/rfdetr/detr.py", line 36, in train
    self.train_from_config(config, **kwargs)
  File "/data/rfdetr/venv/lib/python3.10/site-packages/rfdetr/detr.py", line 74, in train_from_config
    self.model.train(
  File "/data/rfdetr/venv/lib/python3.10/site-packages/rfdetr/main.py", line 405, in train
    shutil.copy2(output_dir / 'checkpoint_best_regular.pth', output_dir / 'checkpoint_best_total.pth')
  File "/home/user/anaconda3/lib/python3.10/shutil.py", line 434, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/user/anaconda3/lib/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'output/checkpoint_best_regular.pth'

Environment

  • rfdetr : 1.0.8
  • os: ubuntu 22.04
  • python: 3.10.9
  • torch: 2.6.0
  • GPU: RTX 6000 Ada

Minimal Reproducible Example

from rfdetr import RFDETRBase

model = RFDETRBase()

model.train(
    dataset_dir='./custom_dataset',
    epochs=5,
    batch_size=16,
    grad_accum_steps=1,
    lr=1e-4,
    output_dir='./output',
    device='cuda:1'
)

The custom dataset structure is as follow:

custom_dataset/
├── test
├── train
└── valid

And in each subset folder, there is the images and _annotations.coco.json

Additional

No response

Are you willing to submit a PR?

  • Yes, I’d like to help by submitting a PR!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 18

Most upvoted comments

Yep, thanks for your quick response! 🚀

Still the same issue despite modifying my annotations as follows and adapting each bounding box annotation accordingly. I’m continuing to investigate and will keep you updated. Let me know if you have any other ideas.

    "categories": [
        {
            "id": 0,
            "name": "Dumb",
            "supercategory": "none"
        },
        {
            "id": 1,
            "name": "class1",
            "supercategory": "Dumb"
        },
        {
            "id": 2,
            "name": "class2",
            "supercategory": "Dumb"
        },
        {
            "id": 3,
            "name": "class3",
            "supercategory": "Dumb"
        },
        {
            "id": 4,
            "name": "class4",
            "supercategory": "Dumb"
        },
        {
            "id": 5,
            "name": "class5",
            "supercategory": "Dumb"
        }
    ]

Image

Here is the terminal logs: rfdetr_logs.txt And the logs from the training: train_logs.txt