numba: Error on import with numpy HEAD

For devs who are trying to fix their environments The problem occurs in the latest version of numpy. Installing a lower version of numpy (<1.24) fixed the problem for us.

Just a heads up that I think I’m hitting this error in CIs now that NumPy 1.24 has been released:

https://github.com/mne-tools/mne-python/actions/runs/3732610574/jobs/6332287508#step:9:56

I expect others might hit it soon, too.

Hello everyone, the solutions posted here did not fix the problem for me. I tried numba v0.56.4 and v0.56.3 and both of these versions did not work with either numpy 1.23.0 or 1.24.0. I installed everything with conda-forge and I am using WSL. Any help would be greatly appreciated. Thank you!

Having the same issue over here with an old numpy version 😕 rolling back to an old version dindt work for me

for me numba=0.56.4 from conda-forge fails locally with numpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I’m doing wrong? @max-sixty, was that with the numba head?

Edit: the install would be

pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgrade

and pip does indeed complain that numba requires numpy<1.24 but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from https://github.com/numba/numba/issues/8615#issuecomment-1323926762 does not apply, point 3 may very well be the cause of the problem.

The PR for review for NumPy 1.24 support is: https://github.com/numba/numba/pull/8691

for me numba=0.56.4 from conda-forge fails locally with numpy=1.24.0.dev0+1120.gf30af6acd, so not sure what I’m doing wrong? @max-sixty, was that with the numba head?

Edit: the install would be

pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre "numpy==1.24.0.dev0+1120.gf30af6acd" --upgrade

and pip does indeed complain that numba requires numpy<1.24 but installs it anyways (as intended in this case, because we do actually want to test with the most recent version, even if that breaks something). So while point 2 from #8615 (comment) does not apply, point 3 may very well be the cause of the problem.

Thanks. This help a lot. I reinstall numpy by “pip install numpy==1.23.0”. Then, the error when “import numba” was solved.

Having this issue with

python=3.8.15, build: h4de0772_0_cpython, Channel: conda-forge numba=0.56.4, build: pypi_0, channel: pypi numpy=1.23.5, build: pypi_0, channel: pypi

what version of numpy should i install?

If you’re using the last released version of Numba (0.56.4 at time of writing), NumPy 1.18 - 1.23 are supported.

Thank you all for your comments/feedback on this issue. This issue seems to have diverged a little from the originally reported problem and so I’m closing this issue as resolved with some specific follow up issues to subscribe to if they are important to you:

1. With respect to the originally posted issue, this comment https://github.com/numba/numba/issues/8615#issuecomment-1324810269 (and some discussion immediately after) explains the decisions surrounding Numba not tracking NumPy HEAD.
2. With respect to the originally posted issue, #8691 is the patch for making Numba compatible with NumPy 1.24.
3. The NumPy 1.24 related problem that manifests as the error:

   SystemError: initialization of _internal failed without raising an exception
   

   is tracked in #8718.

I was having the same problem with the librosa library (The reason was that librosa was working with different versions of scipy and matplotlib), so I uninstalled numpy, matplotlib, and scipy, installed librosa first which installed its own compatible versions of libraries above and then I installed the necessary libraries. That seems to work for me.

Thanks @stuartarchibald — that makes sense. I think we’ll explore excluding anything that runs numba in these tests if this crops up again

@max-sixty no problem, glad it was helpful.

1. There are simply not the resources (human, packages, or compute) for Numba to track NumPy HEAD.

Totally empathize! FWIW, we’ve found that the upstream HEAD tests reduce the maintainer burden, since we can solve problems at a slower pace — no need to rush when there’s an upstream release that breaks something — and we can provide feedback on breakages where there’s a mistaken breaking change. But numba is a more technically complex project than Xarray, so possibly the learning is not transferable.

Thanks for sharing this. I can see that for some projects, aspects of the described approach would certainly be beneficial in terms of maintenance burden.

The Numba maintainers do track changes to dependencies and work with other projects if issues are anticipated, they also often “try out” the code base against newer versions of projects that impact Numba (cPython, LLVM, NumPy) etc to assess the level of effort needed to do the necessary updates. I think the learning from the Numba side is probably, that in developing a compiler, having a stable and predictable environment in which to develop makes it easier to isolate problems. This is in part due to problems often being very niche e.g. one single Python+NumPy+OS+CPU combination might exhibit a particular issue!

The following is also needed because the deprecated MachAr is now removed:

diff --git a/numba/np/arraymath.py b/numba/np/arraymath.py
index 7f53c6563..12edd8e38 100644
--- a/numba/np/arraymath.py
+++ b/numba/np/arraymath.py
@@ -3994,7 +3994,7 @@ def _gen_np_machar():
         return impl
 
 
-_gen_np_machar()
+# _gen_np_machar()
 
 
 def generate_xinfo(np_func, container, attr):

with this I’m able to start running the test suite - as things pass by, everything appears OK so far, but will update here if the test run completes.

…so I’m not sure what we should do from Xarray’s POV — I guess we can try running tests without numba? But that misses some of the utility of these upstream tests.

I’m not sure what is best for the Xarray project, however this information might help informing that decision.

1. Numba’s releases are restricted for use against specific versions of NumPy for reasons as demonstrated in this issue. It is often the case that Numba might “work” on other versions of NumPy, but this cannot be guaranteed as they might not be binary compatible etc.
2. Numba main branch has no restrictions on the version of NumPy, but it also doesn’t track NumPy HEAD. The folks who maintain Numba do an upgrade of the supported NumPy version, often in a single patch, as pre-release packages become available for testing. I’ll raise this issue with the other maintainers at the next triage meeting, but historically the view has been:
   • There are simply not the resources (human, packages, or compute) for Numba to track NumPy HEAD.
   • Development of Numba is considerably easier in stable environments. There are already many variations:
     • Python versions (the bytecode from which Numba compiles changes in literally every version)
     • NumPy versions (algorithms, APIs, etc change in pretty much every version)
     • OS/Architecture (Numba supports at least 7, each with their own individual issues)

Hope this helps?