onnxruntime: [Build] Unable to cross compile for AArch64 Linux
Describe the issue
I am following the instructions here on how to compile onnxruntime v1.15.0 from host x86_64 linux for target aarch64 linux, but the build is failing. I imagine this build is part of your CI pipeline, so probably just the documentation is out of date.
Urgency
medium - high We use onnxruntime for a commercial product and we have deadlines.
Target platform
Host: x86_64 linux (Ubuntu 20.04), Target: aarch64-linux-gnu
Build script
Here are the steps to reproduce:
-
Clone the github repo and submodules, checkout v1.15.0.
-
Ensure we have the cross compiler installed. Output from running this command:
aarch64-linux-gnu-gcc -vis:
Using built-in specs.
COLLECT_GCC=aarch64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/aarch64-linux-gnu/9/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --without-target-system-zlib --enable-libpth-m2 --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu- --includedir=/usr/aarch64-linux-gnu/include
Thread model: posix
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
- Install prebuild protobuf library. Docs say to look at this file / dir to determine version, but it no longer exists in the repo:
Looking through the repo ^ looks like you updated it to version 3.21, so let’s download that version
Download and extract: https://github.com/protocolbuffers/protobuf/releases/download/v21.12/protoc-21.12-linux-x86_64.zip
- Define our cmake toolchain file as follows:
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
SET(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
SET(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
- Create a build directory, run the following cmake command:
cmake -Donnxruntime_GCC_STATIC_CPP_RUNTIME=ON -DCMAKE_BUILD_TYPE=Release -Dprotobuf_WITH_ZLIB=OFF -DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake -Donnxruntime_ENABLE_PYTHON=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_DEV_MODE=OFF -DONNX_CUSTOM_PROTOC_EXECUTABLE=/home/cyrus/work/libs/protoc-21.12-linux-x86_64/bin/protoc ../cmake/
Output:
CMake Deprecation Warning at CMakeLists.txt:21 (cmake_policy):
The OLD behavior for policy CMP0104 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/aarch64-linux-gnu-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Building ONNX Runtime for
-- Performing Test COMPILER_SUPPORT_MF16C
-- Performing Test COMPILER_SUPPORT_MF16C - Failed
F16C instruction set is not supported.
-- Performing Test COMPILER_SUPPORT_FMA
-- Performing Test COMPILER_SUPPORT_FMA - Failed
FMA instruction set is not supported.
-- Performing Test COMPILER_SUPPORT_AVX
-- Performing Test COMPILER_SUPPORT_AVX - Failed
AVX instruction set is not supported.
One or more AVX/F16C instruction flags are not supported.
-- Performing Test onnxruntime_HAVE_BUILTIN_ATOMICS
-- Performing Test onnxruntime_HAVE_BUILTIN_ATOMICS - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test Iconv_IS_BUILT_IN
-- Performing Test Iconv_IS_BUILT_IN - Success
-- Found Iconv: built in to C library
-- Found Patch: /usr/bin/patch
Patch found: /usr/bin/patch
Doing crosscompiling
-- Found Python: /usr/bin/python3.8 (found suitable version "3.8.10", minimum required is "3.8") found components: Interpreter
Loading Dependencies URLs ...
Loading Dependencies ...
CMake Warning at /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src/absl/copts/AbseilConfigureCopts.cmake:70 (message):
Value of CMAKE_SYSTEM_PROCESSOR () is unknown and cannot be used to set
ABSL_RANDOM_RANDEN_COPTS
Call Stack (most recent call first):
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src/CMake/AbseilHelpers.cmake:18 (include)
/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src/CMakeLists.txt:81 (include)
-- Abseil source dir:/home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/abseil_cpp-src
# date: USE_SYSTEM_TZ_DB ON
# date: USE_TZ_DB_IN_DOT OFF
# date: BUILD_SHARED_LIBS OFF
# date: ENABLE_DATE_TESTING OFF
--
-- 3.21.12.0
-- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT
-- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT - Success
-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS
-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success
-- Using the single-header code from /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/nlohmann_json-src/single_include/
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Looking for strtof_l
-- Looking for strtof_l - found
-- Looking for strtoull_l
-- Looking for strtoull_l - found
-- Using toolchain file: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/aarch64.cmake.
-- Found Python: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter
CMake Warning at /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/pytorch_cpuinfo-src/CMakeLists.txt:59 (MESSAGE):
Target processor architecture is not specified. cpuinfo will compile, but
cpuinfo_initialize() will always fail.
-- Found PythonInterp: /usr/bin/python (found version "3.8.10")
-- Using custom protoc executable
Generated: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/onnx-build/onnx/onnx-ml.proto
Generated: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/onnx-build/onnx/onnx-operators-ml.proto
Generated: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp/_deps/onnx-build/onnx/onnx-data.proto
--
-- ******** Summary ********
-- CMake version : 3.26.3
-- CMake command : /home/cyrus/.local/lib/python3.8/site-packages/cmake/data/bin/cmake
-- System : Linux
-- C++ compiler : /usr/bin/aarch64-linux-gnu-g++
-- C++ compiler version : 9.4.0
-- CXX flags : -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -Wnon-virtual-dtor
-- Build type : Release
-- Compile definitions : ORT_ENABLE_STREAM;ORT_NO_RTTI;EIGEN_MPL2_ONLY;_GNU_SOURCE;__STDC_FORMAT_MACROS
-- CMAKE_PREFIX_PATH :
-- CMAKE_INSTALL_PREFIX : /usr/local
-- CMAKE_MODULE_PATH : /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/cmake/external
--
-- ONNX version : 1.14.0
-- ONNX NAMESPACE : onnx
-- ONNX_USE_LITE_PROTO : ON
-- USE_PROTOBUF_SHARED_LIBS : OFF
-- Protobuf_USE_STATIC_LIBS : ON
-- ONNX_DISABLE_EXCEPTIONS : OFF
-- ONNX_WERROR : OFF
-- ONNX_BUILD_TESTS : OFF
-- ONNX_BUILD_BENCHMARKS : OFF
--
-- Protobuf compiler :
-- Protobuf includes :
-- Protobuf libraries :
-- BUILD_ONNX_PYTHON : OFF
Finished fetching external dependencies
-- Performing Test HAS_UNUSED_BUT_SET_PARAMETER
-- Performing Test HAS_UNUSED_BUT_SET_PARAMETER - Success
-- Performing Test HAS_UNUSED_BUT_SET_VARIABLE
-- Performing Test HAS_UNUSED_BUT_SET_VARIABLE - Success
-- Performing Test HAS_UNUSED_VARIABLE
-- Performing Test HAS_UNUSED_VARIABLE - Success
-- Performing Test HAS_CAST_FUNCTION_TYPE
-- Performing Test HAS_CAST_FUNCTION_TYPE - Success
-- Performing Test HAS_PARENTHESES
-- Performing Test HAS_PARENTHESES - Success
-- Performing Test HAS_USELESS_CAST
-- Performing Test HAS_USELESS_CAST - Success
-- Performing Test HAS_NONNULL_COMPARE
-- Performing Test HAS_NONNULL_COMPARE - Success
-- Performing Test HAS_TAUTOLOGICAL_POINTER_COMPARE
-- Performing Test HAS_TAUTOLOGICAL_POINTER_COMPARE - Failed
-- Performing Test HAS_CATCH_VALUE
-- Performing Test HAS_CATCH_VALUE - Success
-- Performing Test HAS_MISSING_BRACES
-- Performing Test HAS_MISSING_BRACES - Success
-- Performing Test HAS_IGNORED_ATTRIBUTES
-- Performing Test HAS_IGNORED_ATTRIBUTES - Success
-- Performing Test HAS_DEPRECATED_COPY
-- Performing Test HAS_DEPRECATED_COPY - Success
-- Performing Test HAS_DEPRECATED_DECLARATIONS
-- Performing Test HAS_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAS_CLASS_MEMACCESS
-- Performing Test HAS_CLASS_MEMACCESS - Success
-- Performing Test HAS_MAYBE_UNINITIALIZED
-- Performing Test HAS_MAYBE_UNINITIALIZED - Success
-- Performing Test HAS_STRICT_ALIASING
-- Performing Test HAS_STRICT_ALIASING - Success
NVCC_ERROR =
NVCC_OUT = No such file or directory
-- Performing Test HAS_AMBIGUOUS_REVERSED_OPERATOR
-- Performing Test HAS_AMBIGUOUS_REVERSED_OPERATOR - Failed
-- Performing Test HAS_DEPRECATED_ANON_ENUM_ENUM_CONVERSION
-- Performing Test HAS_DEPRECATED_ANON_ENUM_ENUM_CONVERSION - Failed
-- Performing Test HAS_UNDEFINED_VAR_TEMPLATE
-- Performing Test HAS_UNDEFINED_VAR_TEMPLATE - Failed
-- Performing Test HAS_FORMAT_TRUNCATION
-- Performing Test HAS_FORMAT_TRUNCATION - Success
-- Performing Test HAS_BITWISE_INSTEAD_OF_LOGICAL
-- Performing Test HAS_BITWISE_INSTEAD_OF_LOGICAL - Failed
-- Performing Test HAS_ENUM_CONSTEXPR_CONVERSION
-- Performing Test HAS_ENUM_CONSTEXPR_CONVERSION - Failed
-- Performing Test HAS_DEPRECATED_BUILTINS
-- Performing Test HAS_DEPRECATED_BUILTINS - Failed
-- Looking for reallocarray
-- Looking for reallocarray - found
-- Found Git: /usr/bin/git (found version "2.25.1")
CMake Warning at CMakeLists.txt:1475 (message):
MPI and NCCL disabled on Win build.
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
CMake Warning (dev) at onnxruntime_mlas.cmake:587:
Syntax Warning in cmake code at column 107
Argument not separated from preceding token by whitespace.
Call Stack (most recent call first):
CMakeLists.txt:1609 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning at onnxruntime_mlas.cmake:63 (message):
AMX instructions NOT supported due to lack of compiler tool chain!
Call Stack (most recent call first):
CMakeLists.txt:1609 (include)
-- Configuring done (22.1s)
-- Generating done (0.2s)
CMake Warning:
Manually-specified variables were not used by the project:
onnxruntime_DEV_MODE
onnxruntime_GCC_STATIC_CPP_RUNTIME
-- Build files have been written to: /home/cyrus/work/c-sdks/3rd_party_libs/onnxruntime/build_tmp
- Run
make, command fails at 70% on this error:
[ 64%] Built target onnxruntime_session
[ 64%] Built target onnxruntime_generate_def
[ 66%] Built target nsync_cpp
[ 69%] Built target re2
[ 69%] Built target clog
[ 70%] Built target cpuinfo
[ 70%] Building C object CMakeFiles/onnxruntime.dir/generated_source.c.o
[ 70%] Linking CXX shared library libonnxruntime.so
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: libonnxruntime_common.a(cpuid_info.cc.o): in function `onnxruntime::CPUIDInfo::ArmLinuxInit()':
cpuid_info.cc:(.text._ZN11onnxruntime9CPUIDInfo12ArmLinuxInitEv+0x42c): undefined reference to `cpuinfo_isa'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: cpuid_info.cc:(.text._ZN11onnxruntime9CPUIDInfo12ArmLinuxInitEv+0x438): undefined reference to `cpuinfo_isa'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: _deps/pytorch_cpuinfo-build/libcpuinfo.a(init.c.o): in function `cpuinfo_initialize':
init.c:(.text.cpuinfo_initialize+0x4): undefined reference to `cpuinfo_arm_linux_init'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: init.c:(.text.cpuinfo_initialize+0x10): undefined reference to `cpuinfo_arm_linux_init'
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: libonnxruntime.so.1.15.0: hidden symbol `cpuinfo_arm_linux_init' isn't defined
/usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/onnxruntime.dir/build.make:153: libonnxruntime.so.1.15.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:2021: CMakeFiles/onnxruntime.dir/all] Error 2
make: *** [Makefile:166: all] Error 2
Error / output
See above box
Visual Studio Version
No response
GCC / Compiler Version
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 23 (9 by maintainers)
for aarch64, adding
to the toolchain file did the trick for me with onnxruntime 1.15.1
Adding
SET(CMAKE_SYSTEM_PROCESSOR aarch64)indeed fixes the issue for v1.15.1. Thank you @mayeut.For anyone else who comes across this issue, here is the full solution:
Download correct version of protocol buffers
Create a file called
aarch64.cmakewith the following contents:Now run the following. Be sure to change the path
ONNX_CUSTOM_PROTOC_EXECUTABLEargument to point to your protoc download:@snnn please be sure to add
SET(CMAKE_SYSTEM_PROCESSOR aarch64)to the documentation, it’s still not there:Feel free to close this issue once you’ve added it to the documentation page.
I was able to cross-compile the ONNX runtime 1.15.1 for ARM 32-bit, but had to make sure no more recent version of Protobuf was installed locally on my PC. For cross-compiling the ONNX runtime 1.11.0 this was however not needed. I think they messed up a bit their CMake build environment, they made some changes in that external deps folder.
I used the following commands (sorry for the hard-coded paths)
This is strange, libcpuinfo.a is in the link command line. But the symbols were still not found. I didn’t write this code. I will try to understand it and reach back to you later.