astropy: import _erfa fails for numpy 1.14

To summarize the slack thread:

• we see this for any downstream CI where coordinates/time/table is imported (as all use erfa).
• it doesn’t matter whether conda-forge or conda default packaged packages are used, both are producing the same issue.
• no issues with pip installed version combinations
• @mwcraig is looking into our conda build.

@dhomeier - thanks for all the sleuthing - at least, we understand now where the problem lies. Remains to understand how best to solve it…

The new conda package seems to work ( at least import astropy._erfa succeeds).

For the future should we run the conda-forge tests against the lowest supported version of numpy? If yes, can do a PR to add that to the conda-forge recipe. It is, I think, one of the only places in our infrastructure where we build, and do some limited testing of, astropy built from the source distribution.

New releases have been made, and uploaded to pypi. The conda-forge packages are also ready, but it’ll take a few days for them to end up on the default channels.

@mwcraig - please do check whether everything is now working as expected (I made the sdists this time with np 1.14.6).

Sounds good. If we have numpy >=1.16 for 4.0, then the problem will go away…

I think then I would go ahead and do another proper releases, that would in turn make the sunpy folks also happy to get their bugfixes released in 3.2.1.

(as, I don’t mind if I need to generate the sdist in an env with older numpy, we just need to add it to the release notes (and then I need to remember, and hope that we can get rid of the old version in the next LTS 😉 ))

Good news: I understand the problem (or at least Bad news: I can’t fix it, it isn’t really a conda problem. The same failure happens if you build from the released tarball. Even more fun, it will never happen when you build from a clean checkout of the source repo, as when wheels are built.

The problem:

The erfa ufunc code ufunc.c is auto-generated from a jinja template. It is regenerated if:

• the template file ufunc.c.templ is newer than the generated file ufunc.c, unless the the files are part of a release (as indicated by astropy/version.py).
• the file ufunc.c does not exist, as is the case in a source checkout.

There is a critical bit of the template at https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 that needs to be included if building for numpy less than 1.16.

The problem is that the check for the numpy version is done at the time the sdist is generated, not at the time the package is compiled.

I’m guessing the sdist was done on a machine with numpy 1.16 or higher. Because of https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 the generated ufunc.c is missing the bit needed to build a working binary on numpy less than 1.16.

If the release had been done a machine with numpy less than 1.16 then I’m guessing that the building the release tarball against numpy 1.16 might fail? No, apparently it does not.

Not sure what the fix is here. The current approach at https://github.com/astropy/astropy/blob/master/astropy/_erfa/ufunc.c.templ#L584 doesn’t work because the numpy version at build time isn’t known.

To reproduce this, grab the source tarball and setup.py install in a an environment with numpy 1.14. Then try import astropy._erfa and you’ll get the error.

Ping @bsipocz @mhvk

It’s obviously somehow related to #8672, only here I believe numpy 1.14.5 is expecting in erfa_ufunc.ab.signature something like “(d3),(d3),(),()->(d3)”, getting “(3),(3),(),()->(3)” instead.

I cannot run any tests, because the collection already fails on the _erfa import:

>>> astropy.test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sw/lib/python3.7/site-packages/astropy/tests/runner.py", line 256, in test
    return runner.run_tests(**kwargs)
  File "/sw/lib/python3.7/site-packages/astropy/tests/runner.py", line 599, in run_tests
    from astropy.table import Table  # pylint: disable=W0611
  File "/sw/lib/python3.7/site-packages/astropy/table/__init__.py", line 41, in <module>
    from .column import Column, MaskedColumn, StringTruncateWarning, ColumnInfo
  File "/sw/lib/python3.7/site-packages/astropy/table/column.py", line 26, in <module>
    from . import groups
  File "/sw/lib/python3.7/site-packages/astropy/table/groups.py", line 7, in <module>
    from .index import get_index_by_names
  File "/sw/lib/python3.7/site-packages/astropy/table/index.py", line 38, in <module>
    from astropy.time import Time
  File "/sw/lib/python3.7/site-packages/astropy/time/__init__.py", line 2, in <module>
    from .formats import *
  File "/sw/lib/python3.7/site-packages/astropy/time/formats.py", line 16, in <module>
    from astropy import _erfa as erfa
  File "/sw/lib/python3.7/site-packages/astropy/_erfa/__init__.py", line 3, in <module>
    from .core import *
  File "/sw/lib/python3.7/site-packages/astropy/_erfa/core.py", line 40, in <module>
    from . import ufunc
ValueError: expect dimension name at position 1 in "(3),(3),(),()->(3)"

But they (almost) all succeed with numpy 1.16.3 installed. BTW as it appears the test for NUMPY_LT_1_16 is done in erfa_generators.py when processing the C source, I am wondering what happens if the package is compiled and built on a system with numpy < 1.16 and later installed along numpy 1.16…