scikit-learn: Scoring functions don't know classes_
Moving the discussion with @amueller from https://github.com/pydata/patsy/issues/77#issuecomment-174335341.
Proposing to:
- add an optional ‘labels’ argument to
log_loss
- add an argument to
make_scorer
to enable passing labels argument to the custom loss function
Related to this - perhaps also allow make_scorer
’s need_threshold = True
to be used for non-binary classification problems. Not sure why it’s limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC
doesn’t have a cheap predict_proba
, but has a cheap decision_function
, so it would allow it to be used with rank-based metris). From make_scorer
perspective, it just seems like a completely arbitrary restriction because it shouldn’t care about what the custom loss function does with the decision_function
output.
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 3
- Comments: 21 (20 by maintainers)
Moving this to 2.0, this needs somebody to pay attention to it though.