scikit-learn: average_precision_score not working as expected
Describe the bug
When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.
Steps/Code to Reproduce
from sklearn.metrics import average_precision_score
# labels
y_true = [1, 1, 1, 1, 1] # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1] # Model's confidence in predictions
# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
Expected Results
An AP score close to 0.
Actual Results
AP score: 1.0
Versions
System:
python: 3.9.19 (main, May 6 2024, 19:43:03) [GCC 11.2.0]
executable: /bin/python
machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31
Python dependencies:
sklearn: 1.5.2
pip: 24.2
setuptools: 75.1.0
numpy: 1.26.4
scipy: 1.13.1
Cython: None
pandas: 1.4.0
matplotlib: 3.5.3
joblib: 1.4.2
threadpoolctl: 3.5.0
Built with OpenMP: True
threadpoolctl info:
user_api: blas
internal_api: openblas
num_threads: 32
prefix: libopenblas
filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
version: 0.3.23.dev
threading_layer: pthreads
architecture: SkylakeX
user_api: blas
internal_api: openblas
num_threads: 32
prefix: libopenblas
filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
version: 0.3.27
threading_layer: pthreads
architecture: SkylakeX
user_api: openmp
internal_api: openmp
num_threads: 32
prefix: libgomp
filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
version: None
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (15 by maintainers)
@glevv It corresponds to the prevalence. If you are in a medical setting, it would be the performance of a dummy classifier that always predict the positive outcome. Basically this is the chance level point. In this setting, you will never reach a precision of 0.0.
We can compare
sklearn.metrics.average_precision_scorewith PyTorch’s versiontorcheval.metrics.functional.binary_auprc.We can define a test function to compare numerical results between scikit-learn and PyTorch:
I ran several test cases, including OP’s, and they all passed. Therefore, if we have some counterexamples along this direction, we will have a better chance of understanding the bug.
I draw it like in my code to compute AP. I tested different method with different interpolations. They all give about 0.5 AP on the example input from above, but the direct sklearn method that is giving 0.97 AP. This is because in all methods the general idea is to compute the area under the PR curve. In my example, all methods stop at recall ~0.53, as my model never goes higher. But scikit-learn assumes a recall of 1 when the threshold is 0, this is not true in object detection.
Another way to just have the plots would be:
AP (Average Precision) is a different metric than Precision. It is the area under the Precision Recall curve: https://scikit-learn.org/1.5/modules/generated/sklearn.metrics.average_precision_score.html
Here is the issue with a less trivial example:
Y_true: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1] Y_scores: [0.9997937083244324, 0.9848889112472534, 0, 0.9497167468070984, 0, 0, 0.9908866882324219, 0.9809730052947998, 0.9870287179946899, 0.9999470710754395, 0, 0, 0.9267389178276062, 0, 0.9746383428573608, 0.9993682503700256, 0.8148384094238281, 0, 0, 0, 0.6392842531204224, 0.9999898672103882, 1.0, 0, 0, 0, 0, 0.9375026822090149, 0.9581012725830078, 0.6106476783752441, 0.9974182844161987, 0, 0, 0, 1.0, 0.999996542930603, 0.8294159173965454, 0.7495031356811523, 0, 0, 0, 0.907082200050354]
If I plot the precision recall curve with this actual data I get:
The area under that curve (so the AP) is about 0.5
But if I plot the curve using the precision/recall used by average_precision_score I get:
Basically it adds an additional point with recall being one and precision as if all predictions were True. Giving a tremendous AP of 0.97. I don’t think this is reasonable. For example, in object detection (where AP is the standard metric), a y_score of 0 means the object was not detected at all. It can never be a positive prediction so a “forced” recall of 1 does not make sense.