scikit-learn: Scoring functions don't know classes_

Moving the discussion with @amueller from https://github.com/pydata/patsy/issues/77#issuecomment-174335341.

Proposing to:

  • add an optional ‘labels’ argument to log_loss
  • add an argument to make_scorer to enable passing labels argument to the custom loss function

Related to this - perhaps also allow make_scorer’s need_threshold = True to be used for non-binary classification problems. Not sure why it’s limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC doesn’t have a cheap predict_proba, but has a cheap decision_function, so it would allow it to be used with rank-based metris). From make_scorer perspective, it just seems like a completely arbitrary restriction because it shouldn’t care about what the custom loss function does with the decision_function output.

About this issue

  • Original URL
  • State: open
  • Created 11 years ago
  • Reactions: 3
  • Comments: 30 (24 by maintainers)

Most upvoted comments

Moving this to 2.0, this needs somebody to pay attention to it though.

Great reminder Ahmed

Dear sklearn team, where are your thoughts on this? If its something you don’t want to pursue, pls just reply confirming that so that we stop chasing you


From: Ahmed Thahir @.> Sent: 26 January 2025 7:19 To: scikit-learn/scikit-learn @.> Cc: nsp121 @.>; Comment @.> Subject: Re: [scikit-learn/scikit-learn] Scoring functions don’t know classes_ (#6231)

Any updates?

— Reply to this email directly, view it on GitHubhttps://github.com/scikit-learn/scikit-learn/issues/6231#issuecomment-2614251303, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5OWWFVGYRV7MPL7VRTEBT2MSD6LAVCNFSM6AAAAABV4E7QW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJUGI2TCMZQGM. You are receiving this because you commented.Message ID: @.***>

So I guess a next step here is to add labels to all scoring functions where it can be relevant?