deepsensor: Potential bug: Incompatibility with NumPy 2.0 leads to AttributeError

Description

The recent upgrade to NumPy 2.0.0 introduces a breaking change due to the removal of np.sctypes. This update has resulted in deepsensor raising an AttributeError upon import when paired with the new version of NumPy. This indicates that the current deepsensor codebase is not compatible with NumPy 2.0.0’s updated API, causing it to break. Indeed, the documentation for NumPy 2.0.0 indicates that np.sctypes has indeed been removed from the namespace:

https://numpy.org/devdocs/numpy_2_0_migration_guide.html#changes-to-namespaces

Here is the full traceback received after trying to import deepsensor.torch:

Traceback (most recent call last):
  File "~/DeepSensor/NumPy-issue/deepsensor_numpy_test.py", line 1, in <module>
    import deepsensor.torch
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/__init__.py", line 20, in <module>
    from .data.processor import DataProcessor
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/__init__.py", line 2, in <module>
    from .loader import TaskLoader
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/loader.py", line 1, in <module>
    from deepsensor.data.task import Task, flatten_X
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/deepsensor/data/task.py", line 5, in <module>
    import lab as B
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/__init__.py", line 9, in <module>
    from .generic import *
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/generic.py", line 11, in <module>
    from .types import (
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/types.py", line 99, in <module>
    Int = Union[tuple([int, Dimension] + np.sctypes["int"] + np.sctypes["uint"])]
  File "~/deepsensor_numpy_test/lib/python3.10/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.sctypes` was removed in the NumPy 2.0 release. Access dtypes explicitly instead.. Did you mean: 'dtypes'?

Sorry to be the bearer of bad news @wesselb, but the error seems to come from the lab package when it tries to access a numpy function. This breaking change, which has also broken lab, could be reported as a separate issue here:

https://github.com/wesselb/lab/issues

This error will affect users who pip install deepsensor using the repository in its current form. One straightforward solution would be pinning the NumPy version in the requirements/requirements*.txt files to the most recent compatible version (i.e. numpy<=1.26.4 or numpy<2).

Reproduction steps

1. Create a new virtual environment for testing (e.g. `deepsensor_numpy_test`, as seen above)
2. pip install `deepsensor` 0.3.6 and its dependencies (including NumPy 2.0.0)
3. Try to import deepsensor code (e.g. `import deepsensor.torch`)
4. Get the Traceback shown above

Version

I am using python 3.10.9. Here is a truncated package list from pip list for completeness:

dask                     2024.6.2
deepsensor               0.3.6
matplotlib               3.9.0
netCDF4                  1.7.1
NeuralProcesses          0.2.6
numpy                    2.0.0
pandas                   2.2.2
pip                      24.1
scikit-learn             1.5.0
scipy                    1.13.1
seaborn                  0.13.2
torch                    2.3.0
xarray                   2024.6.0

The breaking change appears to affect the lab package, which is not pip installed. I found the version number in my virtual environment here: ~/deepsensor_numpy_test/lib/python3.10/site-packages/lab/_version.py, indicating that I’m using the latest version of lab==1.6.5.

OS

Linux

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (1 by maintainers)

Most upvoted comments

I can confirm that the problem seems to be fixed!

My test:

  • Create a new python environment (deepsensor-test-env)
  • pip install deepsensor torch
  • In python3, run import deepsensor.torch

And the code successfully imports! That’s the extent of my testing at the moment. There is a warning that is raised, but it looks unrelated:

/Users/dannes/DeepSensor/deepsensor-test/deepsensor-test-env/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(

@wesselb and @davidwilby, thanks so much for working on this! I’m glad to see this working again. : )

I’ve opened a PR on fdm to hopefully go towards fixing this issue.

@DaniJonesOcean, I’ve digged into this, and I think backends==1.6.6 should be compatible with NumPy 2.0. There is one incompability in a dependency fdm. You can locally hotfix this as follows:

virtualenv -p python3.9 venv
. ./venv/bin/activate
pip install torch neuralprocesses
sed -i 's/np\.math/np/g' ./venv/lib/python3.9/site-packages/fdm/fdm.py
sed -i 's/np\.factorial/math.factorial/g' ./venv/lib/python3.9/site-packages/fdm/fdm.py
sed -i '1s/^/import math\n/' ./venv/lib/python3.9/site-packages/fdm/fdm.py

After this, the following snippet should run correctly:

import numpy as np
import torch

import neuralprocesses.torch as nps

print(np.__version__)  # 2.0.0
print(torch.__version__)  # 2.3.1+cu121

model = nps.construct_convgnp()
pred = model(torch.randn(1, 1, 4), torch.randn(1, 1, 4), torch.randn(1, 1, 10))

# Runs OK!

I will release a fix for fdm soon. For the moment, you can use the above hotfix.

@DaniJonesOcean, that’s great to hear! Thanks again, @davidwilby. 😃 Since the problem seems to be fixed, I think the issue can be closed, so I’ll go ahead and do that. 😃

@DaniJonesOcean Could you try reinstalling in a fresh environment? This looks like something has gone wrong. Line 99, which gives the error, isn’t what it should be (I double checked the package on PyPI).

I’ll do what I can! Maybe other can hop in and assist as well?

Thank you! 😃 Once a little bit of time frees up for me, I should be able to start digging into this too.