scipy: BUG: scipy.test() failures on macOS 12.6/x86_64
Describe your issue.
@rgommers The test suite finishes on arm64:
=================================== 37188 passed, 3336 skipped, 12061 deselected, 138 xfailed, 11 xpassed, 326 warnings in 416.61s (0:06:56) ===================================
but on x86_64 there are failures in “scipy/stats/tests/test_sampling” and eventually pytest dies with “OSError: [Errno 24] Too many open files”:
scipy/stats/tests/test_sampling.py .......................................................................................................................................FFFF [ 97%]
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF.....F.....FFFFFFF.FFFFssFFFssssssssssFFsFF.....F.FF..........F.FFFFssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 97%]
ssssssssssssssssssssssssssssssssssssssFFFFFFFFFF..FFFFFFFFFFFFFFF.........FFFFFFFFF..sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 98%]
sssssssssssssssssssss.FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFssFFFssssssssssFFsFFFFFFFFF..........FFFFFF...
Everything else finishes OK and when I just run ‘scipy.test(tests=[‘scipy/stats/tests’])’ it finishes OK too.
Attached are verbose logs for arm64 and x86_64 as well as the standalone run.
scipy_test_arm64.log scipy_test_x86_64.log scipy_test_stats_x86_64.log
Reproducing Code Example
python3 -c 'import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.test()'
Error message
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/khp/Downloads/Temp/scipy/_lib/_testutils.py", line 69, in __call__
code = pytest.main(pytest_args)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/config/__init__.py", line 167, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/main.py", line 317, in pytest_cmdline_main
return wrap_session(config, _main)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/main.py", line 305, in wrap_session
config.hook.pytest_sessionfinish(
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 55, in _multicall
gen.send(outcome)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/terminal.py", line 818, in pytest_sessionfinish
self.config.hook.pytest_terminal_summary(
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 34, in _multicall
next(gen) # first yield
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/terminal.py", line 833, in pytest_terminal_summary
self.summary_failures()
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/terminal.py", line 1015, in summary_failures
self._outrep_summary(rep)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/terminal.py", line 1034, in _outrep_summary
rep.toterminal(self._tw)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/reports.py", line 90, in toterminal
longrepr_terminal.toterminal(out)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/_code/code.py", line 1030, in toterminal
element[0].toterminal(tw)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/_code/code.py", line 1060, in toterminal
entry.toterminal(tw)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/_code/code.py", line 1147, in toterminal
self._write_entry_lines(tw)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/_code/code.py", line 1137, in _write_entry_lines
tw._write_source(source_lines, indents)
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/_io/terminalwriter.py", line 192, in _write_source
new_lines = self._highlight(source).splitlines()
File "/Users/khp/Library/Python/3.9/lib/python/site-packages/_pytest/_io/terminalwriter.py", line 203, in _highlight
from pygments.formatters.terminal import TerminalFormatter
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 925, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1414, in find_spec
File "<frozen importlib._bootstrap_external>", line 1383, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1347, in _path_importer_cache
OSError: [Errno 24] Too many open files
SciPy/NumPy/Python version information
1.9.3 1.23.4 sys.version_info(major=3, minor=9, micro=6, releaselevel=‘final’, serial=0)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (22 by maintainers)
OK. Good to know in case it comes up again. @tirthasheshpatel if there’s something you can do to reduce the number of open files, please consider it, but it sounds like this doesn’t need to be tracked.
If I understood the above correctly, our current code seems ok so I would be fine closing it and hope it does not pop up later in our CI.
Hi.
This is really referring to things allocated with
malloc, C++new,stdio.fcloseor similar. Essentially any C level operations that you’d have to write out manually if you were writing C code.Python reference counting should happen as normal - it isn’t broken by a custom
__dealloc__method.I’m not quite sure exactly where all the code you’re discussing is, but you have a
cdef class Somethingwhich contains a_messagesattribute which is acdef class MessageStream.MessageStream.__dealloc__closes itself.So
Somethingdoes its manual C deallocation, then it automatically decrefs all it’s PyObject attributes (including_messages). ifMessageStreamhas a reference count of 1 then it is deallocated, andMessageStream.closeshould be called. Obviously if more references to it exist then it’ll stay alive.So I can’t see anything obviously wrong with your destruction scheme here
You might be right. I was relying on Cython that it will call
__dealloc__when ref count of the object reaches zero. But I need someone to confirm that.