scipy: BUG: spatial: long hangs using Qhull in SciPy 1.13.0, in Colour CI

Describe your issue.

Hello,

We have seen our Windows Github Actions build times jumping from 15m to 4h15m after what seems an update from Scipy 1.12.0 to Scipy 1.13.0:

15m: https://github.com/colour-science/colour/actions/runs/8489605040/job/23259884769 4h15m: https://github.com/colour-science/colour/actions/runs/8845206377/job/24288653119

All the functions affected are touching Qhull via our colour.volume.is_within_mesh_volume definition:

2024-04-26T12:28:33.7956840Z ============================= slowest 5 durations =============================
2024-04-26T12:28:33.7957585Z 9020.65s call     colour/volume/tests/test_spectrum.py::TestIsWithinVisibleSpectrum::test_is_within_visible_spectrum
2024-04-26T12:28:33.7958454Z 8888.35s call     colour/volume/spectrum.py::colour.volume.spectrum.is_within_visible_spectrum
2024-04-26T12:28:33.7959562Z 4397.23s call     colour/volume/tests/test_rgb.py::TestRGB_colourspaceVisibleSpectrumCoverageMonteCarlo::test_RGB_colourspace_visible_spectrum_coverage_MonteCarlo
2024-04-26T12:28:33.7960670Z 3785.15s call     colour/volume/rgb.py::colour.volume.rgb.RGB_colourspace_visible_spectrum_coverage_MonteCarlo
2024-04-26T12:28:33.7961663Z 713.57s call     colour/volume/tests/test_pointer_gamut.py::TestIsWithinPointerGamut::test_is_within_pointer_gamut
def is_within_mesh_volume(
    points: ArrayLike, mesh: ArrayLike, tolerance: float = 100 * EPSILON
) -> NDArrayFloat:
    """
    Return whether given points are within given mesh volume using Delaunay
    triangulation.

    Parameters
    ----------
    points
        Points to check if they are within ``mesh`` volume.
    mesh
        Points of the volume used to generate the Delaunay triangulation.
    tolerance
        Tolerance allowed in the inside-triangle check.

    Returns
    -------
    :class:`numpy.ndarray`
        Whether given points are within given mesh volume.

    Examples
    --------
    >>> mesh = np.array(
    ...     [
    ...         [-1.0, -1.0, 1.0],
    ...         [1.0, -1.0, 1.0],
    ...         [1.0, -1.0, -1.0],
    ...         [-1.0, -1.0, -1.0],
    ...         [0.0, 1.0, 0.0],
    ...     ]
    ... )
    >>> is_within_mesh_volume(np.array([0.0005, 0.0031, 0.0010]), mesh)
    array(True, dtype=bool)
    >>> a = np.array([[0.0005, 0.0031, 0.0010], [0.3205, 0.4131, 0.5100]])
    >>> is_within_mesh_volume(a, mesh)
    array([ True, False], dtype=bool)
    """

    triangulation = Delaunay(mesh)

    simplex = triangulation.find_simplex(points, tol=tolerance)
    simplex = np.where(simplex >= 0, True, False)

    return simplex

I saw a reference to a Qhull related change in the mailing list here: https://mail.python.org/archives/list/scipy-dev@python.org/thread/V25I4CIN556ZKKBIA2Q2OLVFGVYVYDEB/ but not the release notes on Github.

Any pointers would be appreciated, we will probably ban this version.

Cheers,

Thomas

Reproducing Code Example

Hard to repro, would need to run Colour's above tests on Windows using 1.13.0.

Error message

N/A

SciPy/NumPy/Python version and system information

1.13.0 vs 1.12.0 on Windows

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 21 (12 by maintainers)

Most upvoted comments

I think that in our use case, it makes sense to disable OMP parallelism globally for the unit tests. I’m not sure where this information should go and if it is already in the documentation but if not, it might be worth considering adding somewhere as this is a gold nugget!

I’m not sure if it’s documented anywhere. I know about it because I was bitten by this on a previous project. The closest docs page is probably Debugging linear algebra related issues. There’s also the upstream documentation, although that’s a little light on why you’d want to restrict the thread count.

Hey @nickodell,

Thanks a ton! Confirming that setting the environment variable worked:

Status
Success
Total duration
[1m 34s](https://github.com/KelSolaar/qhull-regression/actions/runs/9814531816/usage)

I think that in our use case, it makes sense to disable OMP parallelism globally for the unit tests. I’m not sure where this information should go and if it is already in the documentation but if not, it might be worth considering adding somewhere as this is a gold nugget!

Hello @KelSolaar,

Would you mind trying setting the environment variable OMP_NUM_THREADS=1 within your Windows CI?

Let me explain why I think this might help.

I tried running your example on my Windows computer, and I noticed something odd - I had limited pytest to 2 cores with pytest -n 2, but Task Manager said it was using 100% of available CPU. This can sometimes indicate that a program is using thread parallelism in addition to the process-level parallelism provided by xdist.

To check this, I tried setting OMP_NUM_THREADS to 1, and it ran much faster:

(qhull) PS C:\Users\Nick\programming\qhull-regression> pytest -n 2
================================================= test session starts =================================================
platform win32 -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
rootdir: C:\Users\Nick\programming\qhull-regression
configfile: pyproject.toml
plugins: xdist-3.6.1
2 workers [2 items]
..                                                                                                               [100%]
================================================= 2 passed in 13.69s ==================================================
(qhull) PS C:\Users\Nick\programming\qhull-regression> $env:OMP_NUM_THREADS=1
(qhull) PS C:\Users\Nick\programming\qhull-regression> pytest -n 2
================================================= test session starts =================================================
platform win32 -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
rootdir: C:\Users\Nick\programming\qhull-regression
configfile: pyproject.toml
plugins: xdist-3.6.1
2 workers [2 items]
..                                                                                                               [100%]
================================================== 2 passed in 4.98s ==================================================
(qhull) PS C:\Users\Nick\programming\qhull-regression>

Alternatively, if setting OMP_NUM_THREADS to 1 globally is not an option, an alternative would be to use threadpoolctl, which is a library which can use context managers to turn thread-level parallelism on and off within particular context managers. This could be used to selectively disable parallelism within particular tests, or even within parts of your library where parallelism is known to not help.

Example of one of your tests adapted to use threadpoolctl:

from qhull_regression.common import *
from threadpoolctl import threadpool_limits


def test_RGB_colourspace_volume_coverage_MonteCarlo():
    with threadpool_limits(limits=1, user_api='blas'):
        np.testing.assert_allclose(
            RGB_colourspace_volume_coverage_MonteCarlo(int(10e3)),
            47.25326992,
        )

By changing both of your tests to limit thread-level parallelism, I get similar performance to the OMP_NUM_THREADS option.

Could you try one of those, and report whether that helps?

They’re on PyPI, like all beta/RC releases usually are. You can install with pip install scipy --pre.