scikit-learn: Scoring functions don't know classes_
Moving the discussion with @amueller from https://github.com/pydata/patsy/issues/77#issuecomment-174335341.
Proposing to:
- add an optional ‘labels’ argument to
log_loss - add an argument to
make_scorerto enable passing labels argument to the custom loss function
Related to this - perhaps also allow make_scorer’s need_threshold = True to be used for non-binary classification problems. Not sure why it’s limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC doesn’t have a cheap predict_proba, but has a cheap decision_function, so it would allow it to be used with rank-based metris). From make_scorer perspective, it just seems like a completely arbitrary restriction because it shouldn’t care about what the custom loss function does with the decision_function output.
About this issue
- Original URL
- State: open
- Created 11 years ago
- Reactions: 3
- Comments: 30 (24 by maintainers)
Moving this to 2.0, this needs somebody to pay attention to it though.
Great reminder Ahmed
Dear sklearn team, where are your thoughts on this? If its something you don’t want to pursue, pls just reply confirming that so that we stop chasing you
From: Ahmed Thahir @.> Sent: 26 January 2025 7:19 To: scikit-learn/scikit-learn @.> Cc: nsp121 @.>; Comment @.> Subject: Re: [scikit-learn/scikit-learn] Scoring functions don’t know classes_ (#6231)
Any updates?
— Reply to this email directly, view it on GitHubhttps://github.com/scikit-learn/scikit-learn/issues/6231#issuecomment-2614251303, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5OWWFVGYRV7MPL7VRTEBT2MSD6LAVCNFSM6AAAAABV4E7QW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUGI2TCMZQGM. You are receiving this because you commented.Message ID: @.***>
So I guess a next step here is to add
labelsto all scoring functions where it can be relevant?