astropy: import _erfa fails for numpy 1.14
On 3.2rc1 with numpy 1.14.5 installed import astropy.table
fails in _erfa
boiling down to this ufunc
import error:
Python 3.7.3 (default, May 5 2019, 04:25:55)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy.table
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/sw/lib/python3.7/site-packages/astropy/table/__init__.py", line 41, in <module>
from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
File "/sw/lib/python3.7/site-packages/astropy/table/column.py", line 26, in <module>
from . import groups
File "/sw/lib/python3.7/site-packages/astropy/table/groups.py", line 7, in <module>
from .index import get_index_by_names
File "/sw/lib/python3.7/site-packages/astropy/table/index.py", line 38, in <module>
from astropy.time import Time
File "/sw/lib/python3.7/site-packages/astropy/time/__init__.py", line 2, in <module>
from .formats import *
File "/sw/lib/python3.7/site-packages/astropy/time/formats.py", line 16, in <module>
from astropy import _erfa as erfa
File "/sw/lib/python3.7/site-packages/astropy/_erfa/__init__.py", line 3, in <module>
from .core import *
File "/sw/lib/python3.7/site-packages/astropy/_erfa/core.py", line 40, in <module>
from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"
With numpy 1.16.3 this succeeds, but I did not find any note that NUMPY_LT_1_16
support has already been removed.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 36 (36 by maintainers)
To summarize the slack thread:
@dhomeier - thanks for all the sleuthing - at least, we understand now where the problem lies. Remains to understand how best to solve it…
The new conda package seems to work ( at least
import astropy._erfa
succeeds).For the future should we run the conda-forge tests against the lowest supported version of numpy? If yes, can do a PR to add that to the conda-forge recipe. It is, I think, one of the only places in our infrastructure where we build, and do some limited testing of, astropy built from the source distribution.
New releases have been made, and uploaded to pypi. The conda-forge packages are also ready, but it’ll take a few days for them to end up on the default channels.
@mwcraig - please do check whether everything is now working as expected (I made the sdists this time with np 1.14.6).
Sounds good. If we have numpy >=1.16 for 4.0, then the problem will go away…
I think then I would go ahead and do another proper releases, that would in turn make the sunpy folks also happy to get their bugfixes released in 3.2.1.
(as, I don’t mind if I need to generate the sdist in an env with older numpy, we just need to add it to the release notes (and then I need to remember, and hope that we can get rid of the old version in the next LTS 😉 ))
Good news: I understand the problem (or at least Bad news: I can’t fix it, it isn’t really a conda problem. The same failure happens if you build from the released tarball. Even more fun, it will never happen when you build from a clean checkout of the source repo, as when wheels are built.
The problem:
The erfa ufunc code
ufunc.c
is auto-generated from a jinja template. It is regenerated if:ufunc.c.templ
is newer than the generated fileufunc.c
, unless the the files are part of a release (as indicated byastropy/version.py
).ufunc.c
does not exist, as is the case in a source checkout.There is a critical bit of the template at https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 that needs to be included if building for numpy less than 1.16.
The problem is that the check for the numpy version is done at the time the sdist is generated, not at the time the package is compiled.
I’m guessing the sdist was done on a machine with numpy 1.16 or higher. Because of https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 the generated
ufunc.c
is missing the bit needed to build a working binary on numpy less than 1.16.If the release had been done a machine with numpy less than 1.16 then I’m guessing that the building the release tarball against numpy 1.16 might fail? No, apparently it does not.
Not sure what the fix is here. The current approach at https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 doesn’t work because the numpy version at build time isn’t known.
To reproduce this, grab the source tarball and setup.py install in a an environment with numpy 1.14. Then try
import astropy._erfa
and you’ll get the error.Ping @bsipocz @mhvk
It’s obviously somehow related to #8672, only here I believe numpy 1.14.5 is expecting in
erfa_ufunc.ab.signature
something like “(d3),(d3),(),()->(d3)”, getting “(3),(3),(),()->(3)” instead.I cannot run any tests, because the collection already fails on the _erfa import:
But they (almost) all succeed with numpy 1.16.3 installed. BTW as it appears the test for
NUMPY_LT_1_16
is done inerfa_generators.py
when processing the C source, I am wondering what happens if the package is compiled and built on a system with numpy < 1.16 and later installed along numpy 1.16…