scipy: BUG: Importing scipy.optimize break highspy

Describe your issue.

Importing scipy.optimize breaks highspy.

Reproducing Code Example

❯ python                                                                                                                                                                                                                           ─╯
Python 3.11.9 (main, Oct 20 2024, 12:00:32) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy import optimize as opt
>>> import highspy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tomaskrejci/Developer/ampiato/abackend/cvxpy_env/lib/python3.11/site-packages/highspy/__init__.py", line 1, in <module>
    from ._core import (
ImportError: generic_type: type "ObjSense" is already registered!

Error message

ImportError: generic_type: type "ObjSense" is already registered!

SciPy/NumPy/Python version and system information

>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.15.0 2.2.1 sys.version_info(major=3, minor=11, micro=9, releaselevel='final', serial=0)
/Users/tomaskrejci/Developer/ampiato/abackend/cvxpy_env/lib/python3.11/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
  warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "clang",
      "linker": "ld64",
      "version": "15.0.0",
      "commands": "cc"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.11",
      "commands": "cython"
    },
    "c++": {
      "name": "clang",
      "linker": "ld64",
      "version": "15.0.0",
      "commands": "c++"
    },
    "fortran": {
      "name": "gcc",
      "linker": "ld64",
      "version": "13.3.0",
      "commands": "gfortran"
    },
    "pythran": {
      "version": "0.17.0",
      "include directory": "../../../../../../private/var/folders/95/0ydz4d79163427j3k5crp3fh0000gn/T/pip-build-env-2v7lgf9n/overlay/lib/python3.11/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "aarch64",
      "family": "aarch64",
      "endian": "little",
      "system": "darwin"
    },
    "build": {
      "cpu": "aarch64",
      "family": "aarch64",
      "endian": "little",
      "system": "darwin"
    },
    "cross-compiled": false
  },
  "Build Dependencies": {
    "blas": {
      "name": "Accelerate",
      "found": true,
      "version": "unknown",
      "detection method": "extraframeworks",
      "include directory": "unknown",
      "lib directory": "unknown",
      "openblas configuration": "unknown",
      "pc file directory": "unknown"
    },
    "lapack": {
      "name": "Accelerate",
      "found": true,
      "version": "unknown",
      "detection method": "extraframeworks",
      "include directory": "unknown",
      "lib directory": "unknown",
      "openblas configuration": "unknown",
      "pc file directory": "unknown"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.13.6",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/private/var/folders/95/0ydz4d79163427j3k5crp3fh0000gn/T/cibw-run-wwkmuytj/cp311-macosx_arm64/build/venv/bin/python",
    "version": "3.11"
  }
}

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 21 (12 by maintainers)

Most upvoted comments

@rgommers I built scipy using the current main branch and the information given in https://docs.scipy.org/doc/scipy/building/index.html and installed pyomo and highspy. I can now import scipy and import highspy afterward without an error, and also I can import scipy and afterward solve an optimization problem using Highs via pyomo. Thank you very much for fixing this problem so fast!

It is not a cvxpy bug. All they do is this, which is perfectly valid:

from scipy import optimize
import highspy

We don’t want to introduce such foot guns; no other package should have to care what SciPy internals look like.

@musicinmybrain I reported this in #22868 .

This also fixed a bunch of issues we were seeing in Pyomo, thank you!

Thanks for the report @adbuerger. A fix is coming, we already just merged https://github.com/scipy/HiGHS/pull/71

This is a bug, we should avoid this problem And it’s not a regular symbol clash, but an error coming from Pybind11. The explanation can be found here: https://pybind11.readthedocs.io/en/stable/advanced/classes.html#module-local-class-bindings

A ready-made solution is offered by those docs: “To get around this, you can tell pybind11 to keep the external class binding localized to the module by passing the py::module_local() attribute into the py::class_ constructor:”

So let’s see if we can use that - ObjSense seems to be a simple enum, so no need for it to be global in HiGHS. Of course, there may be more such issues.

We had one earlier such issue in SciPy with another pybind11-using module, see fix in gh-19751.