Nuitka: Windows+MSVC+scipy>=1.8.0 → segfault
Let’s consider sample program and nuitka call for compilation on Windows with MSVC: https://github.com/belonesox/bug-nuitka-scipy-180-crash
- pipenv-based
setup-demo.batfor setup pipenv (pls, change your python location onset-python-path.bat). demo_build.batfor build- trivial
test.py
For linux — OK. For scipy <= 1.7.3 → OK
For scipy >=1.8.0RC1 → test.exe crashed on
import scipy.stats or from scipy.stats import hmean.
Unhandled exception at 0x00D11CBA in test.exe: 0xC0000005: Access violation reading location 0x00000010.
[Inline Frame] test.exe!ATTACH_TRACEBACK_TO_EXCEPTION_VALUE(_object * exception_tb, _traceback *) Line 704
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\include\nuitka\exceptions.h(704)
test.exe!impl_scipy$_lib$_docscrape$$$function__14___init__(const Nuitka_FunctionObject * self, _object * * python_pars) Line 4935
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.scipy._lib._docscrape.cpp(4935)
test.exe!CALL_FUNCTION_WITH_ARGS3(_object * called, _object * const * args) Line 1755
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersCalling2.c(1755)
test.exe!CALL_METHOD_WITH_ARGS3(_object * source, _object * attr_name, _object * const * args) Line 10935
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersCalling2.c(10935)
test.exe!impl_scipy$_lib$_docscrape$$$function__42___init__(const Nuitka_FunctionObject * self, _object * * python_pars) Line 24421
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.scipy._lib._docscrape.cpp(24421)
test.exe!Nuitka_CallMethodFunctionPosArgs(const Nuitka_FunctionObject * function, _object * object, _object * const * args, int args_size) Line 2376
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\CompiledFunctionType.cpp(2376)
test.exe!CALL_FUNCTION_WITH_SINGLE_ARG(_object * called, _object * arg) Line 594
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersCalling2.c(594)
test.exe!impl_scipy$stats$_axis_nan_policy$$$function__6__axis_nan_policy_factory$$$function__3_axis_nan_policy_decorator(const Nuitka_FunctionObject * self, _object * * python_pars) Line 6567
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.scipy.stats._axis_nan_policy.cpp(6567)
test.exe!CALL_FUNCTION_WITH_SINGLE_ARG(_object * called, _object * arg) Line 359
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersCalling2.c(359)
test.exe!modulecode_scipy$stats$_stats_py(_object * module, const Nuitka_MetaPathBasedLoaderEntry * loader_entry) Line 160886
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.scipy.stats._stats_py.cpp(160886)
test.exe!loadModule(_object * module, _object * module_name, const Nuitka_MetaPathBasedLoaderEntry * entry) Line 975
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(975)
test.exe!_EXECUTE_EMBEDDED_MODULE(_object * module, _object * module_name, const char * name) Line 1014
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1014)
[Inline Frame] test.exe!EXECUTE_EMBEDDED_MODULE(_object * module) Line 1087
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1087)
test.exe!_path_unfreezer_exec_module(_object * self, _object * args, _object * kwds) Line 1423
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1423)
[External Code]
python38.dll![Frames below may be incorrect and/or missing, no symbols loaded for python38.dll]
test.exe!CALL_FUNCTION_WITH_ARGS5(_object * called, _object * const * args) Line 3240
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersCalling2.c(3240)
test.exe!IMPORT_MODULE5(_object * module_name, _object * globals, _object * locals, _object * import_items, _object * level) Line 134
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersImport.c(134)
test.exe!modulecode_scipy$stats(_object * module, const Nuitka_MetaPathBasedLoaderEntry * loader_entry) Line 852
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.scipy.stats.cpp(852)
test.exe!loadModule(_object * module, _object * module_name, const Nuitka_MetaPathBasedLoaderEntry * entry) Line 975
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(975)
test.exe!_EXECUTE_EMBEDDED_MODULE(_object * module, _object * module_name, const char * name) Line 1014
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1014)
[Inline Frame] test.exe!EXECUTE_EMBEDDED_MODULE(_object * module) Line 1087
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1087)
test.exe!_path_unfreezer_exec_module(_object * self, _object * args, _object * kwds) Line 1423
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1423)
[External Code]
test.exe!CALL_FUNCTION_WITH_ARGS5(_object * called, _object * const * args) Line 3240
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersCalling2.c(3240)
test.exe!IMPORT_MODULE5(_object * module_name, _object * globals, _object * locals, _object * import_items, _object * level) Line 134
at C:\Users\stas\.virtualenvs\nuitka-scipy-stats-crash-vzkNlNj9\Lib\site-packages\nuitka\build\static_src\HelpersImport.c(134)
test.exe!modulecode___main__(_object * module, const Nuitka_MetaPathBasedLoaderEntry * loader_entry) Line 8768
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.__main__.cpp(8768)
test.exe!loadModule(_object * module, _object * module_name, const Nuitka_MetaPathBasedLoaderEntry * entry) Line 975
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(975)
test.exe!_EXECUTE_EMBEDDED_MODULE(_object * module, _object * module_name, const char * name) Line 1014
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1014)
[Inline Frame] test.exe!IMPORT_EMBEDDED_MODULE(const char *) Line 1068
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MetaPathBasedLoader.cpp(1068)
test.exe!EXECUTE_MAIN_MODULE(const char * module_name) Line 416
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MainProgram.cpp(416)
test.exe!wmain(int argc, wchar_t * * argv) Line 1101
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\static_src\MainProgram.cpp(1101)
[External Code]
test.exe!modulecode_scipy$stats$_continuous_distns(_object * module, const Nuitka_MetaPathBasedLoaderEntry * loader_entry) Line 448047
at C:\projects\bugs\nuitka\nuitka-scipy-stats-crash\test.build\module.scipy.stats._continuous_distns.cpp(448047)
Line 4935 of module.scipy._lib._docscrape.cpp
ATTACH_TRACEBACK_TO_EXCEPTION_VALUE(exception_keeper_value_1, exception_keeper_tb_1);
corresponds to 148 line in scipy\_lib\_docscrape.py
try:
self._parse() # ←←←← here
except ParseError as e:
e.docstring = orig_docstring
raise
All generated sources attached → test-build.zip
(another version … test-build.zip )
Crash not depens:
- nuitka version (reproduced on all since 0.7.0, including factory, but factory got another fail → build.log, not tested before 0.7.0)
- Python version (reproduced on w32 python 3.8.10, w64 python 3.10.0)
Not tested — mingw GCC.
I can provide access for VM with Windows, if it helps…
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (19 by maintainers)
It is incredibly slow to compile with MinGW64, what is relatively acceptable with MSVC, takes hours to complete with it. I am doing it, because I have hopes for better tracebacks. But I guess, my plan on this one is to make sure all CPython test suites run with 0.8 and debug python versions, in vague hopes of catching it under defined conditions with reference counting tests.
I thought I reproduced on Linux, but then on investigation it never happened again. These corruption bugs are very susceptible to what the memory released ends up becoming used for next time, and this depends on allocation patterns. Windows memory allocator is more often exposing issues of Nuitka, while not making it easy to debug.
Regarding pipenv install, I am getting complaints that it doesn’t work for branches to update them, pipenv will complain after I push over commits, I guess in the lock file, the commit is is stored, or hash of archive mismatches, something like that. The installing from branch with pipenv ought to work, but I am not sure if it is working or not actually. I have no idea what
{editable = true, ref = "factory", git = "https://github.com/Nuitka/Nuitka.git"}means to pipenv. Does it address the issue with it changing that I was talking about?Supporting pipenv installation is important to me. I love the tool. Make a new report if there is anything wrong. I have to admit, I don’t have test coverage of it.
I think this may well be the same root cause as #1532, and I may just have found it. It seems I missed an optimization that copying of dictionaries does, and that for empty directories, the Python3 split table appears to be there, but it’s not allowed to use it like that. With many copy calls of
kwargswhich since some point is statically known to be of dictionary type, a reference to the dictionary keys is lost, and for larger programs that copy star arguments, it is crashing.I have of course to verify it, but this seems great news potentially.