nncf: NNCF v2.5.0 accuracy and performance degradation of EfficientDet
I tried NNCF 2.5.0 for my effdet model and observed accuracy and performance degradation.
Even if I use pot-backend (use_pot=True), the accuracy and performance is degraded compared to nncf 2.4.0.
The issue is reproduced with the efficientdet-d0 which was exported by the official procedure.
The throughput was measured by benchmark-app of openvino-dev=2023.0.0.
| model | coco_precision | Throughput |
|---|---|---|
| v2.4.0 | 30.68% | 67.14 FPS |
| commit ca54404 (pot backend) | 30.50% | 65.40 FPS |
| commit ca54404 | 24.53% | 60.92 FPS |
I attaches the reproduction code and environments.
I hope the issue will be fixed so that we can migrate to the newer NNCF.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 16
@l-bat Sorry for the confusion. It should have been named
ms_coco_detection_90_class_without_background. While looking for a working config, the name seems to have been inappropriate. I didn’t use the --definitions option of accuracy-checker, so the name wouldn’t affect the behavior I believe.