scikit-learn: CalibratedClassifierCV doesn't interact properly with Pipeline estimators

I am also interested in knowing the status of interplay between RandomizedSearchCV (or GridSearchCV) and CalibratedClassifierCV. I currently do hyper-parameter optimization with RandomizedSearchCV (which by default fits the best model to the entire dataset it was given) and then calibrate this model with CalibratedClassifierCV using cv = 'prefit' argument.

For anyone else reading this, I got it to work with model__base_estimator__max_depth as I guess the parameter is called base_estimator here.

It doesn’t seem to work with fit_params though.

fit_params = {
    'model__base_estimator__sample_weight': np.random.random(size=X.shape[0])
}

search = RandomizedSearchCV(my_pipeline, param_grid, cv=3, n_iter=4, iid=False)
search.fit(X, y, **fit_params)

TypeError: fit() got an unexpected keyword argument 'base_estimator__sample_weight

I guess it would go in this section https://scikit-learn.org/stable/modules/compose.html#nested-parameters

I think they are asking how to do this scenario.

from sklearn.datasets import make_moons
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import RandomizedSearchCV
from sklearn.calibration import CalibratedClassifierCV

X, y = make_moons()

my_pipeline = Pipeline([
    ('imputer', SimpleImputer()),
    ('model', CalibratedClassifierCV(RandomForestClassifier(n_estimators=10), cv=5))
])

param_grid = {
    'model__max_depth': list(range(2, 10))
}

search = RandomizedSearchCV(my_pipeline, param_grid, cv=3, n_iter=4, iid=False)
search.fit(X, y)

How do I modify param_grid so max_depth gets passed to RandomForestClassifier?

This is similar to how VotingClassifier works, but there the estimators parameter takes a tuple so you can give RandomForestClassifier a name and reference it in param_grid.

rf = RandomForestClassifier(n_estimators=10)

my_pipeline = Pipeline([
    ('imputer', SimpleImputer()),
    ('model', VotingClassifier([('rf', rf)]))  # we can name it 'rf'
])

param_grid = {
    'model__rf__max_depth': list(range(2, 10))
}

search = RandomizedSearchCV(my_pipeline, param_grid, cv=3, n_iter=4, iid=False)
search.fit(X, y)
search.best_params_

I think the stackoverflow you cite is mostly a misunderstanding of syntax. If you want us to properly understand your issue here, please provide code to explain what you’re doing / trying.

I’m currently running into the same issue.

Is this clearly an input-validation-issue, so that the encoding of categorical data won’t be attempted before an error is thrown? Is there a way to disable this input-validation locally?

(I am also not familiar with _CalibratedClassifier and how to use it)

So this wasn’t in my example above, but the pipeline gets put into GridSearchCV and then I want to calibrate the model chosen from GridSearchCV. How does GridSearchCV interact with CalibratedClassifierCV?

From my limited understanding, CalibratedClassifierCV produces K models, where each model is constructed with a K-1 folds, and then averages predictions from each of the K models to make a prediction. This seems semantically different from standard CV where you select the model with the best performance over the K folds but then construct a single model using the entire training data. I’m not sure how to include the CalibrationCV into that procedure.

The thing that made sense to me with CalibratedClassifierCV was to prefit the model from GridSearchCV and then calibrate with a different set of data (as the per the docs recommendation).

I should have noted that in my actual use case, I get a different error

ValueError: could not convert string to float:

which is caused by a column of strings in the DataFrame that I pass. In both cases, it is an error about the input data.