scipy: tests with `multiprocessing` hang on Python 3.8 on macOS

I’m running the whole test suite on macOS on Python 3.8 (via Github Actions on my local machine (Catalina)). I’m seeing several issues with multiprocessing, such as #11827.

They’re related to https://bugs.python.org/issue38501. The tests that hang are:

integrate.tests.test__quad_vec.test_quad_vec_pool

I think the best thing to do at the moment is to use the skipif decorator (possibly also coupled to Darwin):

    @pytest.mark.skipif(multiprocessing.get_start_method() != 'fork',
                        reason=('multiprocessing with spawn method is not'
                                ' compatible with pytest.'))

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (16 by maintainers)

Commits related to this issue

test: Explicitly import multiprocessing See https://github.com/scipy/scipy/issues/11835 — committed to rht/mesa by rht 2 years ago
test: Explicitly import multiprocessing See https://github.com/scipy/scipy/issues/11835 — committed to rht/mesa by rht 2 years ago
test: Explicitly import multiprocessing to prevent batch_run hang See https://github.com/scipy/scipy/issues/11835 — committed to rht/mesa by rht 2 years ago
MAINT: GHA MacOS setup.py update * the `setup.py` MacOS Python `3.8` CI job is starting to fail with timeouts for multiprocessing `Pool`-related code in recent PRs like gh-17777 and gh-17829 * nothi... — committed to tylerjereddy/scipy by tylerjereddy a year ago
MAINT, CI: GHA MacOS setup.py update * the `setup.py` MacOS Python `3.8` CI job is starting to fail with timeouts for `multiprocessing` `Pool`-related code in recent PRs like gh-17777 and gh-17829 *... — committed to tylerjereddy/scipy by tylerjereddy a year ago
MAINT, CI: GHA MacOS setup.py update * the `setup.py` MacOS Python `3.8` CI job is starting to fail with timeouts for `multiprocessing` `Pool`-related code in recent PRs like gh-17777 and gh-17829 *... — committed to tylerjereddy/scipy by tylerjereddy a year ago

Most upvoted comments

@peterbell10, you’re correct. Just tested out on local machine.

just the presence of import multiprocessing is enough to prevent hanging (freeze_support not required).
the tests don’t hang if there’s no import of multiprocessing and runtests.py uses parallelisation, I used -j 4.

It’s a bit crufty that just the presence of the import is enough…

andyfaff on Apr 12, 2020

Pure speculation, but is it possible that adding import multiprocessing to runtests.py was what made the difference? It might change how the python module is initialised on the spawned process.

peterbell10 on Apr 12, 2020