numba: LLVM14 linux-aarch64 blocker

This issue is to document a blocker for using LLVM14 on linux-aarch64.

LLVM14 on Linux-aarch64 seems to have a bug (in probably MCJIT+RuntimeDyLd) that affect fails to load exporting symbols and/or setting up GOT info.

Symptom:

The following tests fails:

numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest.test_ex_nested_list                   
numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest.test_ex_inferred_list_jit             
numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest.test_ex_inferred_list                 
numba.tests.doc_examples.test_typed_dict_usage.DocsTypedDictUsageTest.test_ex_typed_dict_njit               
numba.tests.doc_examples.test_typed_dict_usage.DocsTypedDictUsageTest.test_ex_typed_dict_from_cpython       
numba.tests.doc_examples.test_typed_dict_usage.DocsTypedDictUsageTest.test_ex_inferred_dict_njit            
numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest.test_unbalanced_example              
numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest.test_chunksize_with                  
numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest.test_ex_initial_value_dict_compile_time_consts
numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest.test_ex_jitclass_type_hints

They all segfault when executing JITed callback functions that are invoked by externally compiled functions. Most of the cases can be workaround by making the callback functions have internal linkage. However, ChunksizeExamplesTest failures cannot use this workaround due to requirement of how ufunc works in Numba.

A minimal reproducer for the ChunksizeExamplesTest problems is provided in this gist. Observations in the reproducer:

the bug can somehow be avoided by doing O0 on the problematic JIT function.
saving the problematic LLVM IR to file and running it in a fresh process works without problem.

I don’t know what to make off the first observation. The second observation seems to indicate the problem only occurs when multiple LLVM modules are loaded into MCJIT.

The above tests all works in LLVM11.

Related issues:

https://github.com/numba/numba/issues/8567 but this time is in linux only; OSX is fine.
https://github.com/JuliaLang/julia/issues/42295

Future fixes:

https://github.com/JuliaLang/julia/pull/43664#issuecomment-1025304316 answers if OrcJIT+JITLink does not support aarch64-elf as of early 2022.
GSoC 2022 has work that add support for aarch64-elf to jitlink: https://blog.llvm.org/posts/2022-12-21-compiler-research-internships/

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (18 by maintainers)

Commits related to this issue

llvmdev recipe: Add patch that clears GOTOffsetMap See: - https://github.com/llvm/llvm-project/issues/61402 - https://github.com/numba/numba/issues/8738 Fixes numba/numba#8738. — committed to gmarkall/llvmlite by gmarkall a year ago
Add fix for llvm/llvm-project#61402 This addresses an issue that is triggered by Numba 0.57 - see https://github.com/numba/numba/issues/8738 — committed to gmarkall/llvmdev-feedstock by gmarkall a year ago
Add fix for llvm/llvm-project#61402 This addresses an issue that is triggered by Numba 0.57 - see https://github.com/numba/numba/issues/8738 — committed to gmarkall/llvmdev-feedstock by gmarkall a year ago
Add fix for llvm/llvm-project#61402 This addresses an issue that is triggered by Numba 0.57 - see https://github.com/numba/numba/issues/8738 Cherry-Picked-By: H. Vetinari <h.vetinari@gmx.com> — committed to h-vetinari/llvmdev-feedstock by gmarkall a year ago
Add fix for llvm/llvm-project#61402 This addresses an issue that is triggered by Numba 0.57 - see https://github.com/numba/numba/issues/8738 Cherry-Picked-By: H. Vetinari <h.vetinari@gmx.com> — committed to h-vetinari/llvmdev-feedstock by gmarkall a year ago
Add fix for llvm/llvm-project#61402 This addresses an issue that is triggered by Numba 0.57 - see https://github.com/numba/numba/issues/8738 Cherry-Picked-By: H. Vetinari <h.vetinari@gmx.com> — committed to h-vetinari/llvmdev-feedstock by gmarkall a year ago
Add fix for llvm/llvm-project#61402 (#209) This addresses an issue that is triggered by Numba 0.57 - see https://github.com/numba/numba/issues/8738 — committed to conda-forge/llvmdev-feedstock by gmarkall a year ago

Most upvoted comments

With this patch applied to LLVM 14.0.6:

diff --git a/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp b/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
index 3c7f4ec47eb8..205ee5273b27 100644
--- a/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
+++ b/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp
@@ -2407,6 +2407,7 @@ Error RuntimeDyldELF::finalizeLoad(const ObjectFile &Obj,
   }

   GOTSectionID = 0;
+  GOTOffsetMap.clear();
   CurrentGOTIndex = 0;

   return Error::success();

with Numba and llvmlite main following the LLVM 14 merge:

(numballvm14) gmarkall@gm-agx:~/numbadev/numba$ python runtests.py numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest -v -m
Parallel: 9. Serial: 3
test_ex_literal_list (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_ex_initial_value_list_compile_time_consts (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_ex_literal_dict_compile_time_consts (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_ex_inferred_list_jit (numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest) ... ok
test_ex_jitclass (numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest) ... ok
test_ex_inferred_list (numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest) ... ok
test_ex_jitclass_type_hints (numba.tests.doc_examples.test_jitclass.DocsJitclassUsageTest) ... ok
test_ex_nested_list (numba.tests.doc_examples.test_typed_list_usage.DocsTypedListUsageTest) ... ok
test_ex_initial_value_dict_compile_time_consts (numba.tests.doc_examples.test_literal_container_usage.DocsLiteralContainerUsageTest) ... ok
test_chunksize_manual (numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest) ... ok
test_chunksize_with (numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest) ... ok
test_unbalanced_example (numba.tests.doc_examples.test_parallel_chunksize.ChunksizeExamplesTest) ... ok

----------------------------------------------------------------------
Ran 12 tests in 95.065s

OK

All tests listed in the issue description pass. The runtime is huge, but I have a debug build of LLVM with assertions enabled, and LTO disabled for llvmlite (using LTO ICEd the old version of GCC on my Jetson install 😆 ).

gmarkall on Mar 13, 2023