gluonts: Predictions are way too high when modeling an intermittent count series with DeepAR and NegBin distribution?
I’m trying to model a simulated series of weekly seasonal intermittent sales, with values between 0 and 4. I generated 5 years of simulated data:

I trained a DeepAR model with the output distribution set to Negative Binomial (all other settings were the default settings), on 3 years, and generated predictions for the next two. I got the following results (plotting the [70.0, 80.0, 95.0] predictions intervals):

Increasing number of training epochs doesn’t change anything, the loss falls to its lowest value around the 8th to 10th epoch and hovers more or less around there, whether I train for 10 or 100 epochs. I thought training on 3 years and testing on 2 might be too ambitious, so I tried 4y/1y split instead, and the results got much worse - and downright strange - this time with values climbing into the 100s, even though the largest historical value the series ever reaches is 4 (I’m using the same input series, but is seems flat now because the scale is completely skewed by how large the predictions are):

I’m wondering if I am doing anything wrong? Are there any special settings for DeepAR when applied to intermittent series?
For comparison, the DeepAREstimator worked pretty well out of the box for more traditional series (using Student’s distribution), for example:

Details:
Train data: [{‘start’: Timestamp(‘2014-01-05 00:00:00’, freq=‘W-SUN’), ‘target’: array([1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 1., 1., 1., 1., 1., 2., 0., 0., 1., 2., 2., 1., 4., 1., 2., 1., 0., 0., 2., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0., 1., 2., 1., 2., 0., 1., 1., 2., 3., 2., 2., 1., 1., 3., 4., 1., 1., 0., 0., 3., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0., 1., 1., 0., 2., 1., 1., 0., 1., 0., 1., 2., 2., 1., 2., 3., 3., 1., 2., 2., 0., 0., 2., 0., 3., 0., 1., 2., 0., 1., 1.], dtype=float32), ‘source’: SourceContext(source=‘list_data’, row=1)}]
Test data: {‘start’: Timestamp(‘2017-01-08 00:00:00’, freq=‘W-SUN’), ‘target’: array([2., 1., 2., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 1., 0., 1., 2., 3., 1., 0., 3., 2., 1., 0., 0., 2., 2., 2., 1., 0., 2., 0., 2., 2., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 2., 0., 0., 4., 1., 2., 2., 1., 3., 1., 2., 1., 2., 1., 2., 3., 3., 1., 2., 0.], dtype=float32), ‘source’: SourceContext(source=‘list_data’, row=1)}
Estimator used:
estimator = DeepAREstimator(freq="W", prediction_length=105, trainer=Trainer(epochs=10),distr_output=NegativeBinomialOutput()) predictor = estimator.train(training_data=training_data)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (8 by maintainers)
Commits related to this issue
- Fix negative binomial's scaling Fixes issue #718 Also might explain issue #636 — committed to kashif/gluon-ts by kashif 4 years ago
I’m also training for count series and the loss wasn’t converging. I’ve seen this issue and found the PR https://github.com/awslabs/gluon-ts/pull/719 from @kashif. Since it wasn’t merged yet, I forked the fixed distribution and trained again. Now my loss is converging.
Sorry, never mind - I forgot that I have included an additional covariate with missing values which seems to be causing the problem. After removing it things seem to be working fine! Would your recommendation be to impute such variables before training?
I also checked that yesterday after I pulled the master branch but I haven’t observed a particular improvement TBH… but perhaps I wasn’t able to fully reproduce my last trials, so don’t quote me on this 😉
I had a bit of a break from using gluon and now I’m coming back to it again, so I’ll try to report again in case I find something.
I’m not yet a python expert so I’m not 100% sure how to go about that, but in principle yes - that answers the question!