scikit-learn: [BUG] `roc_auc_score` is wrong (edge case)

Describe the bug

Hi,

It seems that on the edge case when there are equal scores, roc_auc_score actually makes a wrong comuptation:

>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.

On would expect either 0.5 or 0.25 depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is speculated by myself, I didn’t go through any of sklearn code).

roc_auc_bug

Any feedback is welcome. Cheers!

Steps/Code to Reproduce

from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])

Expected Results

0.25 or 0.5

Actual Results

0.375

Versions

1.5.0

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 23 (13 by maintainers)

Most upvoted comments

I thought of something that might benefit the discussion: the probabilistic interpretation of AUC.

Consider random variables $X_0$ (resp. $X_1$), corresponding to the score of a uniformly random negative (resp. positive) sample, drawn from the dataset used to plot the ROC curve. In the convention where positive instances should have higher scores (e.g. my opening message), the usual AUC interpretation is $\text{AUC} = \mathbb{P}(X_1 > X_0)$ or $\text{AUC} = \mathbb{P}(X_1 \geqslant X_0)$ (Wikipedia is inconsistent on this as for example, they use the former in the first paragraph of Area under the curve and they use the latter in probabilistic interpretation of AUC. It doesn’t matter in the case of continuous probability distributions).

Then, relative to our previous discussions, it can be easily shown that:

  • previous (pessimistic) interpolation yields $\text{AUC} = \mathbb{P}(X_1 > X_0)$;
  • next (optimistic) interpolation yields $\text{AUC} = \mathbb{P}(X_1 \geqslant X_0)$;
  • linear interpolation yields $\text{AUC} = \frac{1}{2}\big(\mathbb{P}(X_1\geqslant X_0) + \mathbb{P}(X_1 > X_0)\big)$.

Thanks. Feel free to open a PR to implement the ROCCH option.

For the record, if someone comes up with a good reference in the literature about previous vs linear w.r.t. the canonical definition of a ROC curve in an ML context, we should definitely consider reopening that argument.

Sure @ogrisel!

>>> labels = [1, 0, 0, 1]
>>> scores = [2, 5, 10, 10]
>>> fpr, tpr, thresholds = sklearn.metrics.roc_curve(labels, scores)
>>> _ = sklearn.metrics.RocCurveDisplay(fpr=fpr, tpr=tpr).plot()
>>> plt.show()

sklearn_roc

It seems I made a small mistake regarding conventions since the curve is “inverted” (the other convention I used can be recovered by negating both labels and scores) but it’s not important.

My takeaway is that:

  • sklearn indeed correctly identifies the unattainable (FPR, TPR) point due to two last scores being the same,
  • sklearn interpolates linearly between every points, which I think is a weird behaviour statistically speaking. In my opinion, it should either interpolate to the previous point (center illustration above) or to the convex hull (right illustration above).

Complementary data :

>>> thresholds
array([11, 10,  5,  2])
>>> fpr
array([0. , 0.5, 1. , 1. ])
>>> tpr
array([0. , 0.5, 0.5, 1. ])