gluonts: Cannot overfit a single example

Description

Given the introduction example for forecasting I am trying to see if I can get the loss to be negative, but can’t even after 1000 epochs - the loss is flat after 50 iterations. This is a sanity test the system is able to memorize a single example. Without passing this sanity check it is difficult to know if there is a bug in the optimizer

To Reproduce

import pandas as pd
from gluonts.dataset.common import ListDataset
from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
from gluonts.mx.trainer import Trainer

url = "https://raw.githubusercontent.com/numenta/NAB/master/data/realTweets/Twitter_volume_AMZN.csv"
df = pd.read_csv(url, header=0, index_col=0)
training_data = ListDataset(
    [{"start": df.index[0], "target": df.value[:"2015-04-05 00:00:00"]}],
    freq = "5min",
)

prediction_length = 1
epochs= 1000
estimator = SimpleFeedForwardEstimator(freq="5min", prediction_length=prediction_length,num_parallel_samples=10,
                                       trainer=Trainer(epochs=epochs,learning_rate=1e-4,minimum_learning_rate=1e-8,
                                                       patience=250,weight_decay=0),
                                       sampling=True,num_hidden_dimensions=[5000],)
predictor = estimator.train(training_data=training_data)

train_forecast_it, train_it = make_evaluation_predictions(
    dataset=training_data,  # test dataset
    predictor=predictor,  # predictor
    num_samples=100,  # number of sample paths we want for evaluation
)
tain_predictor_output_ts_list = list(train_forecast_it)
train_target_gt_ts_list = list(train_it)

for tain_predictor_output_ts, train_target_gt_ts in zip(tain_predictor_output_ts_list,train_target_gt_ts_list):
    train_target_gt_ts[-60:].plot(linewidth=2)
    tain_predictor_output_ts.plot(color='g', prediction_intervals=[50.0, 90.0])
    plt.grid(which='both')
    plt.show()

Error message or code output

 epoch=1000/1000, avg_epoch_loss=4.24]

Environment

Operating system: mac
Python version: 3.7
GluonTS version:
MXNet version:

(Add as much information about your environment as possible, e.g. dependencies versions.)

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (9 by maintainers)

Most upvoted comments

Adapted snippet:

import pandas as pd
import matplotlib.pyplot as plt

from gluonts.dataset.common import ListDataset
from gluonts.evaluation.backtest import make_evaluation_predictions
from gluonts.model.deepar import DeepAREstimator
from gluonts.mx.trainer import Trainer

url = "https://raw.githubusercontent.com/numenta/NAB/master/data/realTweets/Twitter_volume_AMZN.csv"
df = pd.read_csv(url, header=0, index_col=0)
# training_target = list(df[:"2015-04-05 00:00:00"].value)
training_target = list(df[:100].value)

training_data = ListDataset(
    [{"start": df.index[0], "target": training_target}],
    freq="5min",
)

print(f"train ts length = {len(training_target)}")

prediction_length = 6
epochs = 100
batch_size = 32
num_batches_per_epoch = 50

print(f"samples trained on = {epochs * num_batches_per_epoch * batch_size}")

estimator = DeepAREstimator(
    freq="5min",
    prediction_length=prediction_length,
    batch_size=batch_size,
    trainer=Trainer(
        epochs=epochs,
        num_batches_per_epoch=num_batches_per_epoch,
        learning_rate=1e-3,
    ),
)

predictor = estimator.train(training_data=training_data)

train_forecast_it, train_it = make_evaluation_predictions(
    dataset=training_data,  # test dataset
    predictor=predictor,  # predictor
    num_samples=100,  # number of sample paths we want for evaluation
)
train_forecasts = list(train_forecast_it)
train_series = list(train_it)

for f, s in zip(
    train_forecasts, train_series
):
    s[-60:].plot(linewidth=2)
    f.plot(color="g", prediction_intervals=[50.0, 90.0])
    plt.grid(which="both")
    plt.show()

lostella on Jan 8, 2021