scipy: BUG: spatial: long hangs using Qhull in SciPy 1.13.0, in Colour CI
Describe your issue.
Hello,
We have seen our Windows Github Actions build times jumping from 15m to 4h15m after what seems an update from Scipy 1.12.0 to Scipy 1.13.0:
15m: https://github.com/colour-science/colour/actions/runs/8489605040/job/23259884769 4h15m: https://github.com/colour-science/colour/actions/runs/8845206377/job/24288653119
All the functions affected are touching Qhull via our colour.volume.is_within_mesh_volume definition:
2024-04-26T12:28:33.7956840Z ============================= slowest 5 durations =============================
2024-04-26T12:28:33.7957585Z 9020.65s call colour/volume/tests/test_spectrum.py::TestIsWithinVisibleSpectrum::test_is_within_visible_spectrum
2024-04-26T12:28:33.7958454Z 8888.35s call colour/volume/spectrum.py::colour.volume.spectrum.is_within_visible_spectrum
2024-04-26T12:28:33.7959562Z 4397.23s call colour/volume/tests/test_rgb.py::TestRGB_colourspaceVisibleSpectrumCoverageMonteCarlo::test_RGB_colourspace_visible_spectrum_coverage_MonteCarlo
2024-04-26T12:28:33.7960670Z 3785.15s call colour/volume/rgb.py::colour.volume.rgb.RGB_colourspace_visible_spectrum_coverage_MonteCarlo
2024-04-26T12:28:33.7961663Z 713.57s call colour/volume/tests/test_pointer_gamut.py::TestIsWithinPointerGamut::test_is_within_pointer_gamut
def is_within_mesh_volume(
points: ArrayLike, mesh: ArrayLike, tolerance: float = 100 * EPSILON
) -> NDArrayFloat:
"""
Return whether given points are within given mesh volume using Delaunay
triangulation.
Parameters
----------
points
Points to check if they are within ``mesh`` volume.
mesh
Points of the volume used to generate the Delaunay triangulation.
tolerance
Tolerance allowed in the inside-triangle check.
Returns
-------
:class:`numpy.ndarray`
Whether given points are within given mesh volume.
Examples
--------
>>> mesh = np.array(
... [
... [-1.0, -1.0, 1.0],
... [1.0, -1.0, 1.0],
... [1.0, -1.0, -1.0],
... [-1.0, -1.0, -1.0],
... [0.0, 1.0, 0.0],
... ]
... )
>>> is_within_mesh_volume(np.array([0.0005, 0.0031, 0.0010]), mesh)
array(True, dtype=bool)
>>> a = np.array([[0.0005, 0.0031, 0.0010], [0.3205, 0.4131, 0.5100]])
>>> is_within_mesh_volume(a, mesh)
array([ True, False], dtype=bool)
"""
triangulation = Delaunay(mesh)
simplex = triangulation.find_simplex(points, tol=tolerance)
simplex = np.where(simplex >= 0, True, False)
return simplex
I saw a reference to a Qhull related change in the mailing list here: https://mail.python.org/archives/list/scipy-dev@python.org/thread/V25I4CIN556ZKKBIA2Q2OLVFGVYVYDEB/ but not the release notes on Github.
Any pointers would be appreciated, we will probably ban this version.
Cheers,
Thomas
Reproducing Code Example
Hard to repro, would need to run Colour's above tests on Windows using 1.13.0.
Error message
N/A
SciPy/NumPy/Python version and system information
1.13.0 vs 1.12.0 on Windows
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 21 (12 by maintainers)
I’m not sure if it’s documented anywhere. I know about it because I was bitten by this on a previous project. The closest docs page is probably Debugging linear algebra related issues. There’s also the upstream documentation, although that’s a little light on why you’d want to restrict the thread count.
Hey @nickodell,
Thanks a ton! Confirming that setting the environment variable worked:
I think that in our use case, it makes sense to disable OMP parallelism globally for the unit tests. I’m not sure where this information should go and if it is already in the documentation but if not, it might be worth considering adding somewhere as this is a gold nugget!
Hello @KelSolaar,
Would you mind trying setting the environment variable OMP_NUM_THREADS=1 within your Windows CI?
Let me explain why I think this might help.
I tried running your example on my Windows computer, and I noticed something odd - I had limited pytest to 2 cores with
pytest -n 2, but Task Manager said it was using 100% of available CPU. This can sometimes indicate that a program is using thread parallelism in addition to the process-level parallelism provided by xdist.To check this, I tried setting OMP_NUM_THREADS to 1, and it ran much faster:
Alternatively, if setting OMP_NUM_THREADS to 1 globally is not an option, an alternative would be to use threadpoolctl, which is a library which can use context managers to turn thread-level parallelism on and off within particular context managers. This could be used to selectively disable parallelism within particular tests, or even within parts of your library where parallelism is known to not help.
Example of one of your tests adapted to use threadpoolctl:
By changing both of your tests to limit thread-level parallelism, I get similar performance to the OMP_NUM_THREADS option.
Could you try one of those, and report whether that helps?
They’re on PyPI, like all beta/RC releases usually are. You can install with
pip install scipy --pre.