scipy: BUG: Importing scipy.optimize break highspy
Describe your issue.
Importing scipy.optimize breaks highspy.
Reproducing Code Example
❯ python ─╯
Python 3.11.9 (main, Oct 20 2024, 12:00:32) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy import optimize as opt
>>> import highspy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/tomaskrejci/Developer/ampiato/abackend/cvxpy_env/lib/python3.11/site-packages/highspy/__init__.py", line 1, in <module>
from ._core import (
ImportError: generic_type: type "ObjSense" is already registered!
Error message
ImportError: generic_type: type "ObjSense" is already registered!
SciPy/NumPy/Python version and system information
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.15.0 2.2.1 sys.version_info(major=3, minor=11, micro=9, releaselevel='final', serial=0)
/Users/tomaskrejci/Developer/ampiato/abackend/cvxpy_env/lib/python3.11/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
"Compilers": {
"c": {
"name": "clang",
"linker": "ld64",
"version": "15.0.0",
"commands": "cc"
},
"cython": {
"name": "cython",
"linker": "cython",
"version": "3.0.11",
"commands": "cython"
},
"c++": {
"name": "clang",
"linker": "ld64",
"version": "15.0.0",
"commands": "c++"
},
"fortran": {
"name": "gcc",
"linker": "ld64",
"version": "13.3.0",
"commands": "gfortran"
},
"pythran": {
"version": "0.17.0",
"include directory": "../../../../../../private/var/folders/95/0ydz4d79163427j3k5crp3fh0000gn/T/pip-build-env-2v7lgf9n/overlay/lib/python3.11/site-packages/pythran"
}
},
"Machine Information": {
"host": {
"cpu": "aarch64",
"family": "aarch64",
"endian": "little",
"system": "darwin"
},
"build": {
"cpu": "aarch64",
"family": "aarch64",
"endian": "little",
"system": "darwin"
},
"cross-compiled": false
},
"Build Dependencies": {
"blas": {
"name": "Accelerate",
"found": true,
"version": "unknown",
"detection method": "extraframeworks",
"include directory": "unknown",
"lib directory": "unknown",
"openblas configuration": "unknown",
"pc file directory": "unknown"
},
"lapack": {
"name": "Accelerate",
"found": true,
"version": "unknown",
"detection method": "extraframeworks",
"include directory": "unknown",
"lib directory": "unknown",
"openblas configuration": "unknown",
"pc file directory": "unknown"
},
"pybind11": {
"name": "pybind11",
"version": "2.13.6",
"detection method": "config-tool",
"include directory": "unknown"
}
},
"Python Information": {
"path": "/private/var/folders/95/0ydz4d79163427j3k5crp3fh0000gn/T/cibw-run-wwkmuytj/cp311-macosx_arm64/build/venv/bin/python",
"version": "3.11"
}
}
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 21 (12 by maintainers)
@rgommers I built scipy using the current main branch and the information given in https://docs.scipy.org/doc/scipy/building/index.html and installed pyomo and highspy. I can now import scipy and import highspy afterward without an error, and also I can import scipy and afterward solve an optimization problem using Highs via pyomo. Thank you very much for fixing this problem so fast!
It is not a
cvxpybug. All they do is this, which is perfectly valid:We don’t want to introduce such foot guns; no other package should have to care what SciPy internals look like.
@musicinmybrain I reported this in #22868 .
This also fixed a bunch of issues we were seeing in Pyomo, thank you!
Thanks for the report @adbuerger. A fix is coming, we already just merged https://github.com/scipy/HiGHS/pull/71
This is a bug, we should avoid this problem And it’s not a regular symbol clash, but an error coming from Pybind11. The explanation can be found here: https://pybind11.readthedocs.io/en/stable/advanced/classes.html#module-local-class-bindings
A ready-made solution is offered by those docs: “To get around this, you can tell pybind11 to keep the external class binding localized to the module by passing the
py::module_local()attribute into thepy::class_constructor:”So let’s see if we can use that -
ObjSenseseems to be a simple enum, so no need for it to be global in HiGHS. Of course, there may be more such issues.We had one earlier such issue in SciPy with another pybind11-using module, see fix in gh-19751.
cc @HaoZeke