gluonts: UnboundLocalError: local variable 'lv' referenced before assignment

@DayanSiddiquiNXD, this would be a way to cut a validation dataset which does not overlap with the train dataset:

from gluonts.dataset.repository.datasets import get_dataset
from gluonts.dataset.split import OffsetSplitter
dataset = get_dataset("m4_hourly")
train_length = len(next(iter(dataset.train))["target"])
print(train_length) # 700

def vertical_split(dataset, offset_from_end):
    """
    Split a dataset time-wise in a train and validation dataset.
    """
    dataset_length = len(next(iter(dataset))["target"])
    
    split_offset = dataset_length - offset_from_end

    splitter = OffsetSplitter(
        prediction_length=offset_from_end,
        split_offset=split_offset,
        max_history=offset_from_end)
    
    (_, dataset_train), (_, dataset_validation) = splitter.split(dataset)
    return dataset_train, dataset_validation

dataset_train, dataset_validation = vertical_split(dataset.train, 40)

len(dataset_validation[0]["target"]) # 40

len(dataset_train[0]["target"]) #660

Note, that m4_hourly is a square dataset (all time series have the same length). If you do not have a square dataset, you can use the DateSplitter instead.

In case you have multiple time series, another option you have is splitting the dataset horizontally, meaning that you reserve some time series for validation. You could do it like this, assuming your time series are well shuffled:

def horizontal_split(dataset, item_split_ratio):
    """
    Split a dataset item-wise.
    """
    item_ids = [entry["item_id"] for entry in list(dataset)]

    n_train_items = int(len(set(item_ids))* item_split_ratio)

    dataset_in_sample = [
        ts for ts in dataset if int(ts["item_id"]) < n_train_items
    ]  # assuming items are zero indexed

    dataset_out_of_sample = [
        ts for ts in dataset if int(ts["item_id"]) >= n_train_items
    ]

    return dataset_in_sample, dataset_out_of_sample

@lostella For now, why can’t we just do:

import mxnet as mx
from gluonts.dataset.common import ListDataset, Dataset
from gluonts.dataset.loader import TrainDataLoader
from gluonts.model.deepar import DeepAREstimator
from gluonts.transform import Transformation
from functools import partial
from gluonts.mx.batchify import batchify

from typing import Callable, Optional

class ValidationDataLoader(TrainDataLoader):
    def __init__(self,
                dataset: Dataset,
                transform: Transformation,
                batch_size: int,
                stack_fn: Callable,
                n_eval_steps: Optional[int]=1,
                num_workers: Optional[int]=None,
                max_queue_size: Optional[int]=None,
                num_prefetch: Optional[int] = None,
                shuffle_buffer_length: Optional[int]=None
            ) -> None:
        super().__init__(dataset=dataset,
                        transform=transform,
                        batch_size=batch_size,
                        stack_fn=stack_fn,
                        num_batches_per_epoch=n_eval_steps)


dataset = ListDataset(
    [{"start": "2020-01-01", "target": list(range(1000))}],
    freq="1D"
)

estimator = DeepAREstimator(
    prediction_length=10,
    freq="1D",
)


data_loader = ValidationDataLoader(
    dataset=dataset,
    transform=estimator.create_transformation(),
    batch_size=4,
    stack_fn=partial(batchify, ctx=mx.context.cpu()),
    n_eval_steps=1
)

for _ in range(10):
    print(len(list(data_loader))) # 1 1 1 1 1 1 1 1 1 1 

This would also make @kaijennissen’s code run and I can’t think of any disadvantage of this approach compared to how the validation loader is currently defined.

This does not take away from the need to rethink validation, since the validation mechanism applied by using

predictor = estimator.train(
    training_data=dataset.train, validation_data=dataset.test)

is not in line with the users’ expectation.

The problem occurs also on master. The root cause seems appears to be the way ValidationDataLoader extracts batches of data: the data transformation pipeline gets applied with is_train=True (see here) so that the “future” target is included in the data, and the loss associated with it (say, negative log-likelihood) can be computed; however, this also has the consequence that the instance splitter selects a random number of time windows from the validation dataset.

Now, because of the usage of MultivariateGrouper, the validation dataset consists of a single 8-dimensional series, out of which 0 or more “validation” windows gets sampled by the instance sampler. When 0 are sampled, no validation loss is computed and lv never gets assigned.

In fact, the following minimal, univariate example shows the number of elements produced by the ValidationDataLoader to be sometimes 0, sometimes 1.

import mxnet as mx

from gluonts.dataset.common import ListDataset
from gluonts.dataset.loader import ValidationDataLoader
from gluonts.model.deepar import DeepAREstimator

dataset = ListDataset(
    [{"start": "2020-01-01", "target": list(range(50))}],
    freq="1D"
)

estimator = DeepAREstimator(
    prediction_length=10,
    freq="1D",
)

data_loader = ValidationDataLoader(
    dataset=dataset,
    transform=estimator.create_transformation(),
    batch_size=4,
    ctx=mx.context.cpu(),
)

for _ in range(20):
    print(len(list(data_loader)))

This problem will occur any time the ValidationDataLoader is constructed out of a “singleton” dataset (but may happen with some other small number of time series in the dataset).

The solution to this is to rethink how the ValidationDataLoader should behave and how it is defined: not an easy one, but thank you kindly for submitting this @kaijennissen, this motivates us in improving this part.

@DayanSiddiquiNXD #555

Thanks. Replacing the ValidationDataLoader inside the GluonEstimator with the one you proposed seems to fix the issue.