darts: Historical Backtest With Updating Covariates (Tree/ Regression Models)

As a follow up to the discussion in #2421, I want to clarify the methodology of doing a historical backtest when your covariates are being updated in reality. Assuming that in production I have hourly data and I would like to do a 7 day forecast everyday. Hence, my model would be defined like this in production:"

lgbm_model =  LightGBMModel(
    lags = list(range(-24, 0)),
    past_future_covariates = None,
    lags_future_covariates = list(range(0, 168)),
    output_chunk_length = 24,
    n_jobs=-1,
    random_state=42,
    multi_models=True,
    verbose=0,
    force_col_wise=True
)

lgbm_model.fit(
    series = target_series_train,
    future_covariates = future_cov_train
)

lgbm_model.predict(n=168,  series=target_series, future_covariates=future_cov_series)

For every day’s forecast, I want the model to use the future cov for that day to forecast that day, e.g. to forecast day 3’s values, the model should be using the model’s own forecast on day 2 and the future covariates on day 3. Henec, I set the params as below:

target lags = [-24,....,-1]
lags_future_covariates = list(range(0, 168)),
output_chunk_length=24
n in predict = 168

To do an equivalent backtest of that, I am slightly unsure how to structure the covariates correctly. Assuming I am retraining everyday, this is how I have constructed the historical backtest:

lgbm_model =  LightGBMModel(
    lags=list(range(-24, 0)),
    past_future_covariates = None,
    lags_future_covariates=list(range(0, 168)),
    output_chunk_length=24,
    n_jobs=-1,
    random_state=42,
    multi_models=True,
    verbose=0,
    force_col_wise=True
)

start_date = pd.Timestamp("2024-01-08 00:00:00") 
split_date = pd.Timestamp("2024-04-30 23:00:00")
end_date = pd.Timestamp("2024-05-22 23:00:00")
backtest_results_dict = {}

for date in pd.date_range(split_date, end_date - timedelta(days = 7)):
    target_series_train = target_series[start_date: date]
    future_cov_train = future_cov_series[start_date: date]
    
    lgbm_model.fit(
        series = target_series_train,
        future_covariates = future_cov_train
    )
    
    test_start_date = date + relativedelta(hours = 1)
    target_series_lb = test_start_date - timedelta(days = 1)
    target_series_ub = date
    future_cov_series_lb = test_start_date
    future_cov_series_ub = date + timedelta(days = 1)
    forecast_results_7_day = pd.DataFrame()
    
    for day in range(1, 8):
        data_df = raw_data_df[raw_data_df['forecast_horizon'] == day]
        target_series_test = TimeSeries.from_dataframe(data_df[target_col])[target_series_lb: target_series_ub]
        future_cov_series_test = TimeSeries.from_dataframe(data_df[future_cov])[future_cov_series_lb: future_cov_series_ub]
        forecast_results = lgbm_model.predict(24, series=target_series_test, future_covariates=future_cov_series_test).pd_dataframe()
        forecast_results_7_day = pd.concat([forecast_results_7_day, forecast_results])

        #update dates
        target_series_lb += timedelta(days=1)
        target_series_ub += timedelta(days=1)
        future_cov_series_lb += timedelta(days=1)
        future_cov_series_ub += timedelta(days=1)
    forecast_results_7_day.rename(columns = {'act_price': 'new_best_guess'}, inplace=True)
    backtest_results_dict[str(date)] = forecast_results_7_day

I have constructed a raw_data_df dataframe that contains different forecast horizons corresponding to the future covariates that are being updated, e.g. to forecast day 1, I would use rows with forecast_horizon = 1 in the raw_data_df. However, in my implementation, the future covariates that I am passing are different to the ones in the “production” example above since I am only passing the future covariates for that day specifically, i.e. future covariates for day 3 when forecasting day 3, then day 4 when forecasting day 4.

This doesn’t seem correct to me because the production example is taking in all 7 days of future covariates and forecasting the next 7 days at one go. How do I set my future covariates in the backtest to be same as that?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15

Most upvoted comments

Correct, the forecast quality might deteriorate when n > 24 as the model will start to consume its own forecasts to predict the next steps (t0, when predicting t+24). Note that this is not necessarily a problem, and don’t need to be avoided at all cost.

Since the topic of the discussion deviated quite a bit from the original topic, I am going to close the issue. Feel free to reopen it or open a new one if you still have questions.