scikit-learn: Scoring functions don't know classes_

Moving the discussion with @amueller from https://github.com/pydata/patsy/issues/77#issuecomment-174335341.

Proposing to:

  • add an optional ‘labels’ argument to log_loss
  • add an argument to make_scorer to enable passing labels argument to the custom loss function

Related to this - perhaps also allow make_scorer’s need_threshold = True to be used for non-binary classification problems. Not sure why it’s limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC doesn’t have a cheap predict_proba, but has a cheap decision_function, so it would allow it to be used with rank-based metris). From make_scorer perspective, it just seems like a completely arbitrary restriction because it shouldn’t care about what the custom loss function does with the decision_function output.

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Reactions: 3
  • Comments: 21 (20 by maintainers)

Most upvoted comments

Moving this to 2.0, this needs somebody to pay attention to it though.