sktime: [BUG] `update` of forecasting pipelines does not transform model

Update by @fkiraly: the root cause of the below is, probably, forecasting pipelines not performing transform in the update method. Changed the title to reflect this.

Describe the bug

update_predict_single no longer refits the model in v0.10.1

To Reproduce

forecast_horizon=24
cov=0.95
preds, conf_ints = model.update_predict_single(
    y=y_new, X=None, 
    fh=np.arange(1, forecast_horizon+1),
    update_params=True,
    return_pred_int=True, alpha=1-cov
)

Expected behavior

The model should be refitted when setting update_params to True. It seems like the property class _y attribute gets updated with the new observations, but the model is not refitted, and predictions returned are just longer forecasts of the originally trained model.

Additional context

Versions

System: python: 3.8.11 (default, Jan 25 2022, 23:19:55) [Clang 13.0.0 (clang-1300.0.29.3)] executable: /Users/tsangis/Projects/uk-ds-day-ahead-prices-forecast/venv/bin/python machine: macOS-12.0.1-x86_64-i386-64bit

Python dependencies: pip: 22.0.3 setuptools: 58.0.4 sklearn: 1.0.2 sktime: 0.10.1 statsmodels: 0.13.1 numpy: 1.21.1 scipy: 1.7.3 pandas: 1.4.1 matplotlib: 3.5.1 joblib: 1.1.0 numba: 0.55.1 pmdarima: 1.8.4 tsfresh: None

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 22 (1 by maintainers)

Most upvoted comments

@fkiraly Thank you so much! will give it a shot now! You responsiveness is amazing. We need to start contributing to sktime… We have some methods wrapping the evaluate method for backtesting (what update_predict with a CV was supposed to be doing) with pred intervals, plotly visuaisations for easier inspection of longer time series, etc, that I think would benefit other people

atsangarides on Mar 29, 2022

Ouch - I hope this didn’t incur any trading losses 😅 (I just compulsively had to look up the “IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE” part of our BSD license 😁)

I think I found the error - the _update of the TransformedTargetForecaster calls update of the transformers, but not transform! It should be calling, in sequence of the pipeline, trafo1.update, trafo1.transform, trafo2.update, trafo2.transform, and passing results of transform on to the next update and transform. But instead, it updates everything on the untransformed data.

At least, that’s a bug. It would not explain why the updating runs in no time, but it would explain why the output is garbage.

fkiraly on Mar 24, 2022