scikit-learn: Inconsistency in AUC ROC and AUPR API
Describe the bug
When only one class is present on the groundtruth. The function roc_auc_score
throws an ValueError
and exits while the average_precision_score
returns -0.0
. I feel that both functions should return similar output in this case.
Steps/Code to Reproduce
from sklearn.metrics import roc_auc_score, average_precision_score
y_true = [1]*10
prob = [0.49]*10
auc_roc = roc_auc_score(y_true, prob)
aupr= average_precision_score(y_true, prob)
Expected Results
auc_roc = roc_auc_score(y_true, prob) >> throws ValueError: Only one class present in y_true. ROC AUC score is not defined in that case
aupr=-0.0
Actual Results
Both should have similar outputs
Versions
Python dependencies:
sklearn: 1.1.2
pip: 22.0.2
setuptools: 59.6.0
numpy: 1.23.2
scipy: 1.9.1
Cython: None
pandas: 1.4.3
matplotlib: 3.5.3
joblib: 1.1.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
user_api: openmp
internal_api: openmp
prefix: libgomp
filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/sitepackages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
version: None
num_threads: 24
user_api: blas
internal_api: openblas
prefix: libopenblas
filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-742d56dc.
3.20.so
version: 0.3.20
threading_layer: pthreads
architecture: SkylakeX
num_threads: 24
user_api: blas
internal_api: openblas
prefix: libopenblas
filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.1
8.so
version: 0.3.18
threading_layer: pthreads
architecture: SkylakeX
num_threads: 24
user_api: openmp
internal_api: openmp
prefix: libgomp
filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1
version: None
num_threads: 12
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (11 by maintainers)
So I would not consider it a bug since there are no real “wrong” results. We could improve the consistency since that now:
roc_curve
andROCCurveDisplay
will not raise an error but a warningroc_auc_score
will raise an errorprecision_recall_curve
,average_precision_score
and display will raise a warning.Maybe having
roc_auc_score
raising a warning and having the same behaviour as the curve would be OK.I would like to have some thoughts other core-devs before going forward. ping @Micky774 @thomasjpfan @adrinjalali @ogrisel