scipy: Optimize raises "ValueError: `x0` violates bound constraints" for parameters that are within bounds

After upgrading to 1.5.0, I’m experiencing a similar behavior with the SLSQP minimizer. When I reverse the upgrade back to 1.4.1 the issue disappears.

Version info: Python 3.7.7 Numpy 1.18.5 scipy - various versions mentioned Microsoft Windows 10 Enterprise 10.0.18362 Build 18362

With scipy 1.5.1 and Python 3.8.3 on Linux, I am experiencing a similar issue with the SLSQP minimizer as well.

For example, in this case, my initial solution is: [-10.8623 -22.7164 11.3582 -22.7164] The bounds are: Bounds(array([-260.3 , -260.3 , -123.159, -104.635]), array([ 12.35, -12.35, 12.35, -12.35]), keep_feasible=True)

I don’t get the error with scipy 1.4.1

I came across this issue in google as I was having the same problem but with the SLSQP minimizer. I was just using different initial guesses/seeds to work around it, however, I went back to some old code and found that it did not work, when previously it did. The only thing that had changed was that I upgraded scipy to the version 1.5.0, and sure enough downgrading it to 1.4.1 made it work again.

However, when using the example code above I receive the same error for all the versions of scipy that I have tried (the latest versions of each major release from 1.1 onwards).

I’m not sure how helpful these observations are, but I figured there might be the small possibility that it could help for isolating the cause of the issue. Unfortunately my code that throws this error with 1.5.0 but not with 1.4.1 is a bit long and relies on data from an outside file, so I can’t really post it here as an example.

Version Info: Python 3.7.0 Jupyter notebook 5.6.0 Numpy 1.19.0 scipy - various versions mentioned Windows 10 Education Version 1909 OS build 18363.900

Also confirming that I’m getting this bug with SLSQP

@lukasheinrich (and other interested parties) in scipy 1.5 the underlying numerical differentiation function for the minimize methods (such as SLSQP), and optimize.approx_fprime, was changed to scipy.optimize._numdiff.approx_derivative. This is a much more robust and feature rich numerical differentiation routine than previously used.

One repeat issue was that the minimizers were asking the objective to be evaluated outside lower and upper bounds, and for many systems that’s a big issue. This was occurring because the previous numerical differentiation code, when a value was at the very upper bound, would take a forward (differences) step above that upper bound. With 1.5 approx_derivative is supplied with these lower and upper bounds, and it does not take finite difference steps beyond either of the limits. For example a forward step is turned into a backwards step if up against an upper bound.

For this issue: There are various minimizer codes (such as the SLSQP Fortran routine) that are responsible for generating steps towards a solution. They call back to Python to ask for function and gradient evaluation at new x locations. Unfortunately some of those locations seem to be just outside one or more of the bounds, we’re talking one or two ULP in some cases. approx_derivative then complains that the x for which it’s being asked to evaluate gradients is outside the limits.

Note: the keep_feasible arguments for optimize.Bounds only applies to ***, it has no effect on minimize methods. This needs to be documented.

The best solution would be for the external codes (SLSQP a major culprit) to suggest steps that are strictly inside the lower/upper bounds. I am not a Fortran expert, and am not able to delve into the code to try and fix it. If someone has the expertise, it’d be welcome. One possible fix is to clip the suggestedx to be inside the bounds (around about here), but that’s not the cleanest fix.

EDITED to reflect @mdhaber comment.

Also confirmed from another project (scikit-hep/pyhf#1146). Happy to help debug this.

I had a similar problem using slsqp. I found out that I was modifying the variables inside a constraint. For instance, if you use a fun(x) as constraint, and you are changing the ‘x’ inside the fun(x), you mind end up outside the bounds. In my case, inside the constraint, I made a ‘simple’ copy of x (e.g. new_x = x), then changing new_x, which lead to automatically changing x … So I solved my problem with a deep copy of x (e.g. new_x = deepcopy(x))

@andyfaff confirmed #13009 does fix our problem. Your explanation does sync up with my initial gut reaction (must be something with the steps) which was helpful. For reference, we see the warning pop out:

$ python toys_crash.py 
Background-like:  36%|████████████████████████████████████████████████████████████████▊                                                                                                                    | 179/500 [00:03<00:06, 48.58toy/s]/Users/kratsg/.pyenv/versions/pyhf-scipy/lib/python3.8/site-packages/scipy/optimize/optimize.py:275: RuntimeWarning: Values in x were outside bounds during an SLSQP step, clipping to bounds
  warnings.warn("Values in x were outside bounds during an "

Do you know if the fix will slow down the optimization routine at all because of the extra check that it does?

Note: the keep_feasible arguments for optimize.Bounds only applies to optimize.least_squares

I was under the impression it worked for trust-constr. I haven’t checked in detail, but I see the minimize_trustregion_constr function mentioned bounds.keep_feasible. Is it worth a closer look?

I bumped into this too, when upgrading from scipy 1.1 to 1.5.2. I’m on Windows, but it seems Linux users are seeing the same.

For those using slsqp can you give an example which can reproduce the issue?

@andyfaff, here is a simple example:

from scipy.optimize import minimize, Bounds
import numpy as np
import sys

working = True
while working:
    bounds = Bounds(np.array([0.1]), np.array([1.0]))
    n_inputs = len(bounds.lb)
    x0 = np.array(bounds.lb + (bounds.ub-bounds.lb) * np.random.random(n_inputs))
    try:
        minimize(lambda x: np.linalg.norm(x), x0, method='SLSQP', bounds=bounds)
        print('.', end='')
    except:
        ex = sys.exc_info()
        print('\nBang!', ex[0], ex[1])
        working = False
        
x0, bounds