scikit-learn: [BUG] `roc_auc_score` is wrong (edge case)
Describe the bug
Hi,
It seems that on the edge case when there are equal scores, roc_auc_score actually makes a wrong comuptation:
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
On would expect either 0.5 or 0.25 depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.
I tried illustrating this example below (keep in mind that this behaviour is speculated by myself, I didn’t go through any of sklearn code).
Any feedback is welcome. Cheers!
Steps/Code to Reproduce
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
Expected Results
0.25 or 0.5
Actual Results
0.375
Versions
1.5.0
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 23 (13 by maintainers)
I thought of something that might benefit the discussion: the probabilistic interpretation of AUC.
Consider random variables $X_0$ (resp. $X_1$), corresponding to the score of a uniformly random negative (resp. positive) sample, drawn from the dataset used to plot the ROC curve. In the convention where positive instances should have higher scores (e.g. my opening message), the usual AUC interpretation is $\text{AUC} = \mathbb{P}(X_1 > X_0)$ or $\text{AUC} = \mathbb{P}(X_1 \geqslant X_0)$ (Wikipedia is inconsistent on this as for example, they use the former in the first paragraph of Area under the curve and they use the latter in probabilistic interpretation of AUC. It doesn’t matter in the case of continuous probability distributions).
Then, relative to our previous discussions, it can be easily shown that:
previous(pessimistic) interpolation yields $\text{AUC} = \mathbb{P}(X_1 > X_0)$;next(optimistic) interpolation yields $\text{AUC} = \mathbb{P}(X_1 \geqslant X_0)$;linearinterpolation yields $\text{AUC} = \frac{1}{2}\big(\mathbb{P}(X_1\geqslant X_0) + \mathbb{P}(X_1 > X_0)\big)$.Thanks. Feel free to open a PR to implement the ROCCH option.
For the record, if someone comes up with a good reference in the literature about
previousvslinearw.r.t. the canonical definition of a ROC curve in an ML context, we should definitely consider reopening that argument.Sure @ogrisel!
It seems I made a small mistake regarding conventions since the curve is “inverted” (the other convention I used can be recovered by negating both
labelsandscores) but it’s not important.My takeaway is that:
sklearnindeed correctly identifies the unattainable(FPR, TPR)point due to two last scores being the same,sklearninterpolates linearly between every points, which I think is a weird behaviour statistically speaking. In my opinion, it should either interpolate to the previous point (center illustration above) or to the convex hull (right illustration above).Complementary data :