onnxruntime: [Build] OnnxRuntime `--skip_tests` flag doesn't effect `onnxruntime_test_all` target

Describe the issue

Trying to build OnnxRuntime on Jetson device but keep failing on compilation due to dynamic linking against CUDA runtime.

I have tried to disable that target with --skip_tests but it didn’t works

Urgency

ASAP

Target platform

aarch64

Build script

./build.sh --config Release --skip_submodule_sync --parallel --build_shared_lib --build_dir /root/triton/onnxruntime/build --update --build --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --allow_running_as_root --use_tensorrt --tensorrt_home /usr/src/tensorrt --compile_no_warning_as_error --skip_tests --cmake_extra_defines 'CMAKE_CUDA_ARCHITECTURES=53;62;72;87'

Error / output

[100%] Linking CXX executable onnxruntime_test_all
/usr/bin/ld: warning: libnvcudla.so, needed by /usr/local/cuda/lib64/libcudla.so.1, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvSetTaskTimeoutInMs'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvCreateDevice'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvDeviceGetCount'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvDeviceGetAttribute'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvImportExternalMemory'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvGetNvSciSyncAttributes'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvGetExternalExportTable'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvModuleGetAttributes'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvModuleUnload'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvGetExportTable'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvSubmitTask'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvDestroyDevice'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvModuleLoadFromMemory'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvMemRegister'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvMemUnregister'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvGetLastError'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvImportExternalSemaphore'
/usr/bin/ld: /usr/local/cuda/lib64/libcudla.so.1: undefined reference to `cudlaDrvGetVersion'
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:4803: onnxruntime_test_all] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2311: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

Visual Studio Version

No response

GCC / Compiler Version

CMake 3.27.1

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Comments: 21 (21 by maintainers)

Most upvoted comments

Ticket can be closed I was able to build ORT using ARM machine without tests. My mistake was is that i stick to the Mac ARM , which break my build.

Hi @mc-nv I reproduced your issue on my local jetson device, and here’s what I did to fix the missing lib:

  1. If there’s no libnvcudla.so under /usr/local/cuda-12.2/compat: sudo apt-get install -y cuda-compat-12-2
  2. Add this lib path to env: export LD_LIBRARY_PATH="/usr/local/cuda-12.2/lib64:/usr/local/cuda-12.2/compat:$LD_LIBRARY_PATH"

More detail of cuDLA: https://docs.nvidia.com/cuda/cuda-for-tegra-appnote/index.html#cudla

If you want to avoid building the tests, --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF