skpro: [BUG] cyclic boosting - sporadic test failures due to convergence failure

The recently added cyclic boosting estimator sporadically fails tests due to failed convergence of the loss, e.g.,:

FAILED skpro/regression/tests/test_cyclic_boosting.py::test_cyclic_boosting_with_manual_paramaters - cyclic_boosting.utils.ConvergenceError: Your cyclic boosting training seems to be diverging. In the 9. iteration the current loss: 52.52700124396056, is greater than the trivial loss with just mean predictions: 20.816666666666666.

FYI @setoguchi-naoki, @felixwick

About this issue

Original URL
State: open
Created 5 months ago
Comments: 30

Commits related to this issue

[BUG] fix `CyclicBoosting._predict_quantiles` (#195) Fixes `CyclicBoosting` non-conformance described in https://github.com/sktime/skpro/issues/188, by renaming args of `_predict_quantiles`. — committed to sktime/skpro by fkiraly 5 months ago

Most upvoted comments

Don’t forget: The more quantiles you need, the more expensive it gets … at least with the usual pinball loss approaches.

I’m considering a situation where you already have them - they do not necessarily need to be fitted independently.

fkiraly on Feb 9, 2024

Ok, Cyclic Boosting 1.4.0 is there. @setoguchi-naoki will go ahead and make the relevant changes here (He already knows what is to do.). In short:

remove individual QPD creation loop in QPD_S and QPD_B classes (it’s now done directly in Cyclic Boosting)
use original rather than extended J-QPD modes in skpro, and drop QPD_U (reason is that extended modes are not vectorized for now, and the benefit is marginal)

FelixWick on Feb 9, 2024

Slight adaptations will be needed: getting rid of the loop over QPD calls. But we can do that when I’m done in Cyclic Boosting.

FelixWick on Feb 8, 2024

@fkiraly I have found an upstream solution by allowing QPD to take full arrays rather than individual quantile values. I’m confident that will fix this here. Give me some days to work it out and build a new Cyclic Boosting release.

FelixWick on Feb 7, 2024

That should not happen with small data sets. Maybe something is messed up. Let me have a look these days.

I suppose it’s good then that we have stringent tests. FYI, I think we also just found a bug in sklearn: https://github.com/sktime/skpro/pull/192 They do not seem to be testing their probabilistic interfaces systematically!

fkiraly on Jan 30, 2024