scipy: BUG (?): box-cox related `BracketError`-s downstream in `sktime`

Describe your issue.

Since 1.11.0, the BoxCoxTransformer in sktime is failing with a BracketError, which ultimately comes from optimize.brent and optimize.fminbound.

We would appreciate help with diagnosing the issue, as none of the more obvious fixes have helped.

Further details:

• the original code was a copy of boxcox_normmax in scipy. Replacing it with current boxcox_normmax and/or using custom optimize.minimize_scalar does not fix the error, see https://github.com/sktime/sktime/pull/4770.
• the code works without any problems on < 1.11.0
• specifying an initial bracket that should theoretically work (e.g., (1.0, 2.0)) does not seem to help either.

PS: the docstrings in the current main version were not too clear about the “should”, i.e., when should scipy be called and when a de-novo implementation. The PR https://github.com/sktime/sktime/pull/4770 also fixes that.

Reproducing Code Example

With sktime 0.20.0 or earlier versions or current main, and scipy 1.11.0:

from sktime.transformations.series.boxcox import BoxCoxTransformer
from sktime.utils._testing.panel import _make_panel_X

X = _make_panel_X(
    n_instances=7, n_columns=1, n_timepoints=10, random_state=42
)

est = BoxCoxTransformer()

est.fit(X)

sktime bug report: https://github.com/sktime/sktime/issues/4769

Error message

BracketError: The algorithm terminated without finding a valid bracket. Consider trying different initial points.

Full traceback:

---------------------------------------------------------------------------
BracketError                              Traceback (most recent call last)
Cell In[2], line 10
      4 X = _make_panel_X(
      5     n_instances=7, n_columns=1, n_timepoints=10, random_state=42
      6 )
      8 est = BoxCoxTransformer()
---> 10 est.fit(X)

File C:\Workspace\sktime\sktime\transformations\base.py:439, in BaseTransformer.fit(self, X, y)
    436     self._fit(X=X_inner, y=y_inner)
    437 else:
    438     # otherwise we call the vectorized version of fit
--> 439     self._vectorize("fit", X=X_inner, y=y_inner)
    441 # this should happen last: fitted state is set to True
    442 self._is_fitted = True

File C:\Workspace\sktime\sktime\transformations\base.py:1205, in BaseTransformer._vectorize(self, methodname, **kwargs)
   1202     else:
   1203         transformers_ = self.transformers_
-> 1205     self.transformers_ = X.vectorize_est(
   1206         transformers_, method=methodname, **kwargs
   1207     )
   1208     return self
   1210 if methodname in TRAFO_METHODS:
   1211     # loop through fitted transformers one-by-one, and transform series/panels

File C:\Workspace\sktime\sktime\datatypes\_vectorize.py:584, in VectorizedDF.vectorize_est(self, estimator, method, args, args_rowvec, return_type, rowname_default, colname_default, varname_of_self, **kwargs)
    581     args_i[varname_of_self] = group
    583 est_i_method = getattr(est_i, method)
--> 584 est_i_result = est_i_method(**args_i)
    586 if group_name is None:
    587     group_name = rowname_default

File C:\Workspace\sktime\sktime\transformations\base.py:436, in BaseTransformer.fit(self, X, y)
    434 # we call the ordinary _fit if no looping/vectorization needed
    435 if not vectorization_needed:
--> 436     self._fit(X=X_inner, y=y_inner)
    437 else:
    438     # otherwise we call the vectorized version of fit
    439     self._vectorize("fit", X=X_inner, y=y_inner)

File C:\Workspace\sktime\sktime\transformations\series\boxcox.py:156, in BoxCoxTransformer._fit(self, X, y)
    154 X = X.flatten()
    155 if self.method != "guerrero":
--> 156     self.lambda_ = _boxcox_normmax(X, bounds=self.bounds, method=self.method)
    157 else:
    158     self.lambda_ = _guerrero(X, self.sp, self.bounds)

File C:\Workspace\sktime\sktime\transformations\series\boxcox.py:377, in _boxcox_normmax(x, bounds, brack, method)
    374     raise ValueError("Method %s not recognized." % method)
    376 optimfunc = methods[method]
--> 377 return optimfunc(x)

File C:\Workspace\sktime\sktime\transformations\series\boxcox.py:364, in _boxcox_normmax.._mle(x)
    360 def _eval_mle(lmb, data):
    361     # function to minimize
    362     return -boxcox_llf(lmb, data)
--> 364 return optimizer(_eval_mle, args=(x,))

File C:\Workspace\sktime\sktime\transformations\series\boxcox.py:322, in _make_boxcox_optimizer..optimizer(func, args)
    321 def optimizer(func, args):
--> 322     return optimize.brent(func, brack=brack, args=args)

File c:\ProgramData\Anaconda3\envs\sktime-skbase-311\Lib\site-packages\scipy\optimize\_optimize.py:2641, in brent(func, args, brack, tol, full_output, maxiter)
   2569 """
   2570 Given a function of one variable and a possible bracket, return
   2571 a local minimizer of the function isolated to a fractional precision
   (...)
   2637 
   2638 """
   2639 options = {'xtol': tol,
   2640            'maxiter': maxiter}
-> 2641 res = _minimize_scalar_brent(func, brack, args, **options)
   2642 if full_output:
   2643     return res['x'], res['fun'], res['nit'], res['nfev']

File c:\ProgramData\Anaconda3\envs\sktime-skbase-311\Lib\site-packages\scipy\optimize\_optimize.py:2678, in _minimize_scalar_brent(func, brack, args, xtol, maxiter, disp, **unknown_options)
   2675 brent = Brent(func=func, args=args, tol=tol,
   2676               full_output=True, maxiter=maxiter, disp=disp)
   2677 brent.set_bracket(brack)
-> 2678 brent.optimize()
   2679 x, fval, nit, nfev = brent.get_result(full_output=True)
   2681 success = nit < maxiter and not (np.isnan(x) or np.isnan(fval))

File c:\ProgramData\Anaconda3\envs\sktime-skbase-311\Lib\site-packages\scipy\optimize\_optimize.py:2448, in Brent.optimize(self)
   2445 def optimize(self):
   2446     # set up for optimization
   2447     func = self.func
-> 2448     xa, xb, xc, fa, fb, fc, funcalls = self.get_bracket_info()
   2449     _mintol = self._mintol
   2450     _cg = self._cg

File c:\ProgramData\Anaconda3\envs\sktime-skbase-311\Lib\site-packages\scipy\optimize\_optimize.py:2417, in Brent.get_bracket_info(self)
   2415     xa, xb, xc, fa, fb, fc, funcalls = bracket(func, args=args)
   2416 elif len(brack) == 2:
-> 2417     xa, xb, xc, fa, fb, fc, funcalls = bracket(func, xa=brack[0],
   2418                                                xb=brack[1], args=args)
   2419 elif len(brack) == 3:
   2420     xa, xb, xc = brack

File c:\ProgramData\Anaconda3\envs\sktime-skbase-311\Lib\site-packages\scipy\optimize\_optimize.py:3047, in bracket(func, xa, xb, args, grow_limit, maxiter)
   3045     e = BracketError(msg)
   3046     e.data = (xa, xb, xc, fa, fb, fc, funcalls)
-> 3047     raise e
   3049 return xa, xb, xc, fa, fb, fc, funcalls

BracketError: The algorithm terminated without finding a valid bracket. Consider trying different initial points.

SciPy/NumPy/Python version and system information

1.11.0 1.25.0 sys.version_info(major=3, minor=11, micro=3, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SKYLAKEX MAX_THREADS=2
    pc file directory: c:/opt/64/lib/pkgconfig
    version: 0.3.21.dev
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /c/opt/64/include
    lib directory: /c/opt/64/lib
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SKYLAKEX MAX_THREADS=2
    pc file directory: c:/opt/64/lib/pkgconfig
    version: 0.3.21.dev
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.10.4
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.35
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.3.0
  pythran:
    include directory: C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-12ywd98f\overlay\Lib\site-packages/pythran
    version: 0.13.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: windows
Python Information:
  path: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-uqg6lkbl\cp311-win_amd64\build\venv\Scripts\python.exe
  version: '3.11'