scikit-learn: Inconsistency in AUC ROC and AUPR API

Describe the bug

When only one class is present on the groundtruth. The function roc_auc_score throws an ValueError and exits while the average_precision_score returns -0.0. I feel that both functions should return similar output in this case.

Steps/Code to Reproduce

from sklearn.metrics import roc_auc_score,  average_precision_score

y_true = [1]*10
prob = [0.49]*10

auc_roc = roc_auc_score(y_true, prob)
aupr= average_precision_score(y_true, prob)

Expected Results

auc_roc = roc_auc_score(y_true, prob) >> throws ValueError: Only one class present in y_true. ROC AUC score is not defined in that case

aupr=-0.0

Actual Results

Both should have similar outputs

Versions

Python dependencies:                                                                                                
      sklearn: 1.1.2                                                                                                
          pip: 22.0.2                                                                                               
   setuptools: 59.6.0                                                                                               
        numpy: 1.23.2                                                                                               
        scipy: 1.9.1                                                                                                
       Cython: None                                                                                                 
       pandas: 1.4.3                                                                                                
   matplotlib: 3.5.3                                                                                                
       joblib: 1.1.0                                                                                                
threadpoolctl: 3.1.0  

Built with OpenMP: True                                                                                             
                                                                                                                    
threadpoolctl info:          
       user_api: openmp      
   internal_api: openmp      
         prefix: libgomp     
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/sitepackages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0    

version: None        
    num_threads: 24          

       user_api: blas        
   internal_api: openblas    
         prefix: libopenblas                              
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-742d56dc.
3.20.so                      
        version: 0.3.20      
threading_layer: pthreads    
   architecture: SkylakeX    
    num_threads: 24          

       user_api: blas        
   internal_api: openblas    
         prefix: libopenblas                              
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.1
8.so                         
        version: 0.3.18      
threading_layer: pthreads    
   architecture: SkylakeX    
    num_threads: 24          

       user_api: openmp      
   internal_api: openmp      
         prefix: libgomp     
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1        
        version: None        
    num_threads: 12

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (11 by maintainers)

Most upvoted comments

So I would not consider it a bug since there are no real “wrong” results. We could improve the consistency since that now:

  • Calling roc_curve and ROCCurveDisplay will not raise an error but a warning
  • roc_auc_score will raise an error
  • precision_recall_curve, average_precision_score and display will raise a warning.

Maybe having roc_auc_score raising a warning and having the same behaviour as the curve would be OK.

I would like to have some thoughts other core-devs before going forward. ping @Micky774 @thomasjpfan @adrinjalali @ogrisel