sktime: [BUG] Very different behaviour depending on whether make_pipeline is from sktime or sklearn

Describe the bug It seems odd that make_pipeline should produce very different score results. I found this exploring one of the Examples files: https://www.sktime.net/en/latest/examples/classification/channel_selection.html .

To Reproduce Υou can see the difference in results by commenting/uncommenting the appropriate make_pipeline lines below. Results are 0.25 (sklearn) and 0.975 (sktime), suggesting there is some important differences here.

import warnings

warnings.filterwarnings("ignore")

from sklearn.linear_model import RidgeClassifierCV
# Which version of make_pipeline should we use when results differ so much!!?!?!
# from sklearn.pipeline import make_pipeline
from sktime.pipeline import make_pipeline

from sktime.datasets import load_UCR_UEA_dataset
from sktime.transformations.panel import channel_selection
from sktime.transformations.panel.rocket import Rocket

cs = channel_selection.ElbowClassPairwise()  # ECP

rocket_pipeline = make_pipeline(cs, Rocket(), RidgeClassifierCV())

data = "BasicMotions"
X_train, y_train = load_UCR_UEA_dataset(data, split="train", return_X_y=True)
X_test, y_test = load_UCR_UEA_dataset(data, split="test", return_X_y=True)

rocket_pipeline.fit(X_train, y_train)

rocket_pipeline.score(X_test, y_test)

Expected behavior I didn’t expect such a difference in results.

Additional context What happens if you’ve loaded both forms of make_pipeline in one session? Is only the first form active?

Versions

System: python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0] executable: /opt/conda/bin/python3.8 machine: Linux-5.15.0-72-generic-x86_64-with-glibc2.10

Python dependencies: pip: 23.3.2 sktime: 0.25.0 sklearn: 0.24.2 skbase: 0.6.2 numpy: 1.22.4 scipy: 1.6.3 pandas: 1.2.4 matplotlib: 3.4.1 joblib: 1.0.1 numba: 0.53.1 statsmodels: 0.12.2 pmdarima: None statsforecast: None tsfresh: None tslearn: None torch: None tensorflow: None tensorflow_probability: None

About this issue

  • Original URL
  • State: open
  • Created 5 months ago
  • Comments: 27

Most upvoted comments

This might be possible in a few days …