pandas: ValueError: numpy.dtype has the wrong size, try recompiling
I have been setting up a new CI server and I’m hitting a problem when running our tests on the new server - tests which use Pandas are giving this error:
File "/root/.virtualenvs/myproject/local/lib/python2.7/site-packages/pandas/__init__.py", line 7, in <module>
from . import hashtable, tslib, lib
File "pandas/src/numpy.pxd", line 157, in init pandas.lib (pandas/lib.c:78660)
ValueError: numpy.dtype has the wrong size, try recompiling
I have read other threads about the cause of this problem but they don’t seem to apply in my case.
We are running in a virtualenv, inside a Docker container (Debian Stretch).
There is no ‘system’ numpy, pandas or cython installed:
root@e00c968153d1:/# pip freeze | grep -i numpy
root@e00c968153d1:/# pip freeze | grep -i pandas
root@e00c968153d1:/# pip freeze | grep -i cython
In our requirements.txt we have:
numpy==1.9.1
pandas==0.15.2
but…
$ workon myproject
$ ipython
In [1]: import numpy
In [2]: numpy.version.full_version
Out[2]: '1.9.1'
In [3]: import pandas
numpy.dtype has the wrong size, try recompiling
Now, I have found a manual fix, which is to re-install pandas:
$ pip uninstall pandas
$ pip install --no-cache-dir pandas==0.15.2
The need for --no-cache-dir suggests something about the problem, but I don’t understand what exactly. I am installing in a ‘virgin’ docker container, there is no pip cache before I first install requirements.txt:
$ ls -l /root/.cache/pip/wheels
$ ls: cannot access /root/.cache/pip/wheels: No such file or directory
After pip install -r requirements.txt:
$ locate pandas
/root/.cache/pip/wheels/f7/7b/7c/6de5614135da1e25d864beec01ed40bb43fcafdeda870a05f6/pandas-0.15.2-cp27-none-linux_x86_64.whl
/root/.virtualenvs/myproject/lib/python2.7/site-packages/pandas
...
(only one installation of pandas found)
The manual fix is not acceptable as I need to automate the build.
Appreciate if you can suggest other avenues to explore towards a solution.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 1
- Comments: 21 (13 by maintainers)
further showing a dockerfile is not any evidence. this needs to be reproducible on a stock system.
you are compiling numpy with a different version that pandas. Not really sure how you are installing. FYI it is MUCH easier to simply use
conda.@thanatos you will need to provide more output (the full log of the sequence of commands and its output) to be able to say more. But as Jeff already pointed out above, this error means that somehow the installed pandas is compiled against a wrong version of numpy (or later another version of numpy is installed)
as I said this is an issue with you system. it is likely cached.
or pip might be at fault. IOW it tries to complete the install of pandas, then installs numpy, but its not installed so pandas can’t find it. you can try to install numpy first. then with a new pip command install pandas.