darts: [BUG] LightGBM past covariates look-ahead bias

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from darts.datasets import EnergyDataset
from darts.models import LightGBMModel
from darts import TimeSeries

Describe the bug

I am concerned that there is a specific corner case of look-ahead bias. This seems to be the case with the following configuration:

• model == LightGBMModel
• output_chunk_length < horizon
• past_covariates != None

To Reproduce

After loading some arbitraty data, we will generate two versions of past_covariates series:

1. pc that has not been manipulated and consisting of pc_train and pc_test series
2. pc_zeros in which pc_test has been floored to zero

data = EnergyDataset().load().pd_dataframe().head(1000)

y = data['total load actual'].fillna(0)
pc = data['generation biomass'].fillna(0)

length = 400
y_train, y_test = y.iloc[:length], y.iloc[length:]
pc_train, pc_test = pc.iloc[:length], pc[length:]

zeros = np.zeros(600)
zeros = pd.Series(zeros, index=pc_test.index)
pc_zeros = pd.concat([pc_train, zeros])

pc.iloc[390:410].plot(label='pc');
pc_zeros.iloc[390:410].plot(label='pc_zeros');
plt.legend();

# Convert to TimeSeries
pc = TimeSeries.from_series(pc)
y_train = TimeSeries.from_series(y_train)
pc_train = TimeSeries.from_series(pc_train)
pc_test = TimeSeries.from_series(pc_test)
pc_zeros = TimeSeries.from_series(pc_zeros)

Next we train a model where output_chunk_length == horizon. Then we predict 24 steps ahead using both pc and pc_zeros as an input. As expected, this produces identical results since manipulating past_covariates in prediction window should not affect the results.

model = LightGBMModel(
    lags=10, 
    lags_past_covariates=10, 
    output_chunk_length=24,
    random_state=0)

model.fit(series=y_train, past_covariates=pc_train);

y_pred = model.predict(24, past_covariates=pc)
y_pred_noise = model.predict(24, past_covariates=pc_zeros)

y_pred.plot(label='pc')
y_pred_noise.plot(label='pc_zeros')

However, when output_chunk_length < horizon, the results are markedly different:

model = LightGBMModel(
    lags=10, 
    lags_past_covariates=10, 
    output_chunk_length=1,
    random_state=0)

model.fit(series=y_train, past_covariates=pc_train);

y_pred = model.predict(24, past_covariates=pc)
y_pred_noise = model.predict(24, past_covariates=pc_zeros)

y_pred.plot(label='pc')
y_pred_noise.plot(label='pc_zeros')

Conclusion

It seems that there is a look-ahead bias with specific LightGBM configuration. This is somewhat dangerous given that for LightGBMModel the output_chunk_length is an optional parameter defaulting to 1 and there is no warning in the current class documentation:

output_chunk_length (int) – Number of time steps predicted at once by the internal regression model. Does not have to equal the forecast horizon n used in predict(). However, setting output_chunk_length equal to the forecast horizon may be useful if the covariates don’t extend far enough into the future.

Proposed solution

As a quick fix, enforce output_chunk_length==horizon.

System

• Python version: 3.8.13
• Darts version: 0.24.0