statsmodels: BUG: Combining SciPy 1.14.1 and statsmodels 0.14.2 on OSX causes "symbol not found in flat namespace '_npy_cabs'"

Describe the bug

When importing statsmodels.api, the following error appears:

Traceback (most recent call last):
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/test.py", line 1, in <module>
    import statsmodels.api as sm
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/api.py", line 136, in <module>
    from .regression.recursive_ls import RecursiveLS
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/regression/recursive_ls.py", line 14, in <module>
    from statsmodels.tsa.statespace.mlemodel import (
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 33, in <module>
    from .simulation_smoother import SimulationSmoother
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/simulation_smoother.py", line 11, in <module>
    from .kalman_smoother import KalmanSmoother
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/kalman_smoother.py", line 11, in <module>
    from statsmodels.tsa.statespace.representation import OptionWrapper
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/representation.py", line 10, in <module>
    from .tools import (
  File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/tools.py", line 14, in <module>
    from . import (_initialization, _representation, _kalman_filter,
  File "statsmodels/tsa/statespace/_initialization.pyx", line 1, in init statsmodels.tsa.statespace._initialization
ImportError: dlopen(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_npy_cabs'

This issue was originally reported on StackOverflow. I have independently reproduced the failure on OSX, but failed to reproduce it on Linux.

Curiously, this issue only shows up when SciPy 1.14.1 is installed. If SciPy 1.14.0 is installed, this error does not show up. Similarly, if statsmodels 0.14.1 is installed, the issue also goes away. Strangely, downgrading from statsmodels 0.14.2 to 0.14.1 and upgrading back to 0.14.2 also fixes the issue.

I’m unsure if this is a bug in statsmodels or SciPy - it seems very strange that a change in another package could cause this kind of error.

Code Sample, a copy-pastable example if possible

import statsmodels.api as sm

If you are using a released version, have you verified that the bug exists in the main branch of this repository? It helps the limited resources if we know problems exist in the current main branch so that they do not need to check whether the code sample produces a bug in the next release.

I was unable to verify whether this issue has been fixed in main, because building the package from source also makes the issue go away.

For example, if I install scipy and statsmodels like this:

pip install scipy==1.14.1 git+https://github.com/statsmodels/statsmodels@v0.14.2

the error goes away, despite being the same version that breaks when I use pre-built binaries.

Expected Output

I expected that the latest version of SciPy and statsmodels could be used together.

Output of import statsmodels.api as sm; sm.show_versions()

This is the output of import statsmodels.api as sm; sm.show_versions() for the working version of SciPy. After installing SciPy 1.14.1, import statsmodels.api as sm no longer works.

import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS

Python: 3.12.5.final.0 OS: Darwin 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:54 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6030 arm64 byteorder: little LC_ALL: None LANG: en_US.UTF-8

statsmodels

Installed: 0.14.2 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels)

Required Dependencies

cython: Not installed numpy: 2.1.0 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/numpy) scipy: 1.14.0 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/scipy) pandas: 2.2.2 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/pandas) dateutil: 2.9.0.post0 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/dateutil) patsy: 0.5.6 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/patsy)

Optional Dependencies

matplotlib: Not installed cvxopt: Not installed joblib: Not installed

Developer Tools

IPython: Not installed jinja2: Not installed sphinx: Not installed pygments: Not installed pytest: Not installed virtualenv: Not installed

Additional details:

x-ref scipy/scipy#21434

Original report from StackOverflow

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 17 (10 by maintainers)

Commits related to this issue

Most upvoted comments

I’ve done a test of all combinations of SciPy 1.14.0/1.14.1, NumPy 2.0.1/2.1.0, and statsmodels 0.14.1/0.14.2.

The result is that the only failing configuration is latest SciPy plus latest NumPy plus latest statsmodels.

scipy Version statsmodels Version numpy Version Compatibility
1.14.0 0.14.1 2.0.1 incompatible (statsmodels requires numpy < 2)
1.14.0 0.14.1 2.1.0 incompatible (statsmodels requires numpy < 2)
1.14.0 0.14.1 1.26.4 works
1.14.0 0.14.2 2.0.1 works
1.14.0 0.14.2 2.1.0 works
1.14.1 0.14.1 2.0.1 incompatible (statsmodels requires numpy < 2)
1.14.1 0.14.1 2.1.0 incompatible (statsmodels requires numpy < 2)
1.14.1 0.14.1 1.26.4 works
1.14.1 0.14.2 2.0.1 works
1.14.1 0.14.2 2.1.0 fails

Everything else either works, or pip refuses to install that combination of packages.

The source of the issue has been found. It is due to some changes in the arch used by the GitHub MaxOS runners, which were previously x86_64 and have moved to arm64. This means that the x86_64 wheels were not correctly linked. The fix is in place, although there are a number of bugs due to changes in other libraries (probably pandas) that need to be addressed before we can release 0.14.3.

https://github.com/MacPython/statsmodels-wheels/actions/runs/10844514094

There seems to be a decent bug here that isn’t easy to fix. It appears in the 0.14.2 code even if built against SciPy 1.14.1.