numba: Program crashes with ACCESS_VIOLATION. Windows only.

Reporting a bug

>>> import numba
>>> numba.__version__
'0.53.0'

This bug was tested on different Python versions (3.7, 3.9) and different numba versions (0.45.0, 0.53.0, 0.50.0) I am using package called librosa which depends on resampy which utilizes numba. Also I am using native Csound library using ctype bindings.

Whenever I am trying to run JITed code program crashes (resampy.interp.resample_f). If I disable JIT compilation by NUMBA_DISABLE_JIT=1 program works fine. If I remove Csound or librosa programs work fine as well. Bug is Windows only reproducable. Here is bug reproduce steps:

Download latest csound version
Copy ctcsound.py binding from csound/bin folder to site-packages or to bug.py file location
Place any .wav file over bug.py
Run following code:

import ctcsound
import librosa

if __name__ == "__main__":
    ct = ctcsound.Csound()
    audio, orig_sr = librosa.load("some_sample.wav", sr=None)
    audio = librosa.resample(audio, orig_sr, 44020)
    print("NOT CRASHED!")

Program crashes with 0xC0000005 error (ACCESS_VIOLATION). Debugger screenshot It seems that it’s trying to dereference [[_PyThreadState_Current]+0x48], but because [_PyThreadState_Current] is 0 next instruction getting 0x48. 0x48 is outside of virtual memory map that why we are getting 0xC0000005. I think it’s something about GIL.

About this issue

Original URL
State: open
Created 3 years ago
Comments: 35 (15 by maintainers)

Most upvoted comments

I put together the following script to show the addresses of all instances of Py_IsInitialized (a CPython symbol) and what LLVM is seeing:

import libpy_check
import ctcsound
from llvmlite import binding as llvm
import numba  # needed to initialize LLVM

if __name__ == "__main__":
    print('before Csound'.center(80, '-'))
    print('llvm found address of Py_IsInitialized', 
          hex(llvm.address_of_symbol("Py_IsInitialized")))
    libpy_check.check_python()


    ct = ctcsound.Csound()

    print('after Csound'.center(80, '-'))
    libpy_check.check_python()
    print('llvm found address of Py_IsInitialized', 
          hex(llvm.address_of_symbol("Py_IsInitialized")))

The libpy_check is available at https://gist.github.com/sklam/4837581ff47ba72c0ee9678e1aebe1c7.

The output of the script is:

$ python bug.py
---------------------------------before Csound----------------------------------
llvm found address of Py_IsInitialized 0x7ffdec6a3d80
b'C:\\path\\to\\miniconda3\\envs\\iss6862\\python38.dll' 0x7ffdec6a3d80
b'C:\\path\\to\\miniconda3\\envs\\iss6862\\python3.DLL' 0x7ffdec6a3d80
0dBFS level = 32768.0
--Csound version 6.15 (double samples) Aug 13 2020
[commit: 18c2c7897425f462b9a7743cee157cb410c88198]
libsndfile-1.0.29pre1
----------------------------------after Csound----------------------------------
b'C:\\path\\to\\miniconda3\\envs\\iss6862\\python38.dll' 0x7ffdec6a3d80
b'C:\\path\\to\\miniconda3\\envs\\iss6862\\python3.DLL' 0x7ffdec6a3d80
b'C:\\path\\to\\dev\\python27.dll' 0x6099c3b0
llvm found address of Py_IsInitialized 0x6099c3b0
end of score.              overall amps:      0.0
           overall samples out of range:        0
0 errors in performance
Elapsed time at end of performance: real: 0.020s, CPU: 0.020s

This confirms that the line ct = ctcsound.Csound() is loading a python27.dll into memory. From that point on, LLVM is seeing Py_IsInitialized from the python27.dll instead from the python38.dll.

A related note from LLVM at: https://github.com/llvm/llvm-project/blob/62ec4ac90738a5f2d209ed28c822223e58aaaeb7/llvm/lib/Support/Windows/DynamicLibrary.inc#L136-L140 is indicating that the symbol search order is reversed on win32. On Unix, it searches from the oldest to newest. On Win32, it searches from newest to oldest. That explains why this is a windows only problem.

It is still odd that a process can have symbols with duplicated names. How do one tell which dll module is the current interpreter? Can a process have multiple python dll modules of the same version?

sklam on Mar 26, 2021

I think the reason of the issue is found, so you can close issue, but you can create new one exactly for that case.

Redict on Mar 25, 2021

@rawwar some people have that issue, some don’t. I really have no clue what is the reason. I have this script working on my laptop, but on PC it’s crashes.

Redict on Mar 25, 2021

Running into the same crash as @Redict using Python 3.7 and both numba 0.43/0.53

0dBFS level = 32768.0
--Csound version 6.15 (double samples) Aug 13 2020
[commit: 18c2c7897425f462b9a7743cee157cb410c88198]
libsndfile-1.0.29pre1

Process finished with exit code -1073741819 (0xC0000005)

atjessehill on Mar 25, 2021