cvxpy: Install order with numpy can break cvxpy
Describe the bug Installing a pinned version of numpy together with cvxpy in the same pip command will cause cvxpy to be built against the wrong version of numpy. This is presumably due to pip building all packages first before installing, meaning at build time, cvxpy does not know which version of numpy it should build for.
To Reproduce
On a clean python environment (I used the python:3.7 docker image):
$ pip install numpy==1.19.2 cvxpy==1.1.1
...
$ python
Python 3.7.10 (default, Apr 10 2021, 16:02:25)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cvxpy
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
If we instead run a command to install numpy first, it works fine:
$ pip install numpy==1.19.2 && pip install cvxpy==1.1.12
...
$ python
Python 3.7.10 (default, Apr 10 2021, 16:02:25)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cvxpy
>>>
If we instead pin numpy to its latest version, that also works fine:
$ pip install numpy==1.20.3 cvxpy==1.1.1
...
$ python
Python 3.7.10 (default, Apr 10 2021, 16:02:25)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cvxpy
>>>
Presumably, this will break on the next major numpy upgrade.
Expected behavior cvxpy should install correctly in all cases, to allow it to be installed as part of a pinned requirements.txt file.
Version
- OS: MacOS 11.12.3, on docker
- CVXPY Version: 1.1.12
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 26 (20 by maintainers)
Well, it’s not “we”, it’s the maintainers here, and packaging for python is a painful and thankless job. Maybe @SteveDiamond & @rileyjmurray can comment, but unless you prepare a PR that runs & builds the various flavours of manylinux-wheels required, we shouldn’t use “we”.
It does work without, you “just” have to compile. And - I dare say - if you work in the data science space, conda is becoming the default moreso than pip (though depending on which packages you’re using exactly, but certainly if you’re doing GPU-heavy things).
PS. Here’s a (not-so-recent-anymore) report of the maintainers of arrow, why maintaining wheels was becoming an unbearable overhead, whereas conda just works (by comparison). The wheel-building has since been reinstated, but more or less only because some people stepped up to help.
Deserves shout-out: good on you @naphatkrit for taking up the debugging and fixing.
update on my end: it turns out cvxpy behaves correctly, but it is actually scs (a dependency of cvxpy) that does not specify the version of numpy correctly. I’ve put up a PR there instead. https://github.com/bodono/scs-python/pull/38. Perhaps scs should be optional for users of cvxpy who don’t plan to use it?
would still love to get linux wheel though, but it’s not immediately blocking this issue.
It is now much easier than before with cibuildwheel and github actions. I used in in the qdldl-python wrapper and @akshayka used it for the pymde package.