gammapy: Crash after macOS Catalina update in multiprocess call

I updated to macOS Catalina and now I get this crash

gammapy/cube/tests/test_ring.py::TestAdaptiveRingBackgroundEstimator::test_run_const_width libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Fatal Python error: Aborted

libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Fatal Python error: Aborted

from here: https://github.com/gammapy/gammapy/blob/c9ed2c68ed00ae37b14c8092d6096b0f2dbae95a/gammapy/maps/image_utils.py#L51

It occurs every time I run pytest -s -v gammapy/cube/tests/, but never when I run pytest -s -v gammapy/cube/tests/test_ring.py, i.e. it depends on Python process state and likely previous multiprocessing calls.

Log: https://gist.github.com/cdeil/a75211856a3bcab751ead707df9708c9#file-gistfile1-txt-L88

There is https://www.anaconda.com/how-to-restore-anaconda-after-macos-catalina-update/ but I couldn’t find any issue yet concerning multiprocessing issues after Catalina update.

I’ll rm -r my Anaconda install now and re-install from scratch, let’s see … I just wanted to put this issue in the tracker to have a record, and as reference if others run into the same issue.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 15 (9 by maintainers)

Most upvoted comments

Using psutils I found out that this one file is opened several times:

/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf

Possibly the std::runtime_error: Couldn't close file relates to this file.

After

conda uninstall matplotlib

the issue is gone, all tests (including the ones using multiprocessing) pass for me.

I’ll try to debug further next week, see if I can create a minmal test case that only involves matplotlib and multiprocessing.

cdeil on Oct 11, 2019

So https://github.com/matplotlib/matplotlib/issues/15410 is still open, and it’s not 100% clear why the crash is happening. But it is already clear that this is not a bug in the Gammapy code. Depending on viewpoint, one say it’s the fault of Apple, or CPython or matplotlib. matplotlib is keeping a font file open for performance reasons, and then on child process creation tries to close it from the child process. That worked well so far, but apparently now on macOS Catalina it doesn’t any more, which is not totally surprising since a major change in that version was process isolation and security changes, moving macOS towards a locked down execution environment like iOS. I note that at the moment these are still guesses from my side, and the issue might or might not be there for other CPython versions and binary builds.

Closing this issue now, resolved by #2472

cdeil on Oct 17, 2019