onnxruntime: Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != NULL' failed!

My apologies for the ramble. Desperation tends to do that. I had resolved the issue on my own. It turns out that PyInstaller was not including all of the necessary CUDA libraries. Including them manually allowed onnxruntime to start up (and then crash when it couldn’t find the cudnn_*_infer libraries, but that error was transparent.)

I will say though that it is incredibly frustrating to have spent the time on what ended up being a fairly simple issue. I understand that the import hack is done to avoid the eyes of the auditor, but this same hack made it much more difficult to realize that it was just a matter of a missing dependency. I know I’m barking up the wrong tree here since I could’ve just not used Python, but this type of error would’ve happened much sooner in the pipeline and likely had a more useful error message in any compiled language.

In my specific case, after looking at the onnx runtime requirements again, I noticed that I might be missing cudnn. I tried installing libcudnn8 and libcudnn8-dev, and I was able to run my code successfully. Not quite sure if libcudnn8-dev was necessary or if libcudnn8 would have been sufficient.

If you are running with TensorrtExecutionProvider, reinstall libnvinfer libraries solves the issue in my case:

• Uninstall libnvinfer libraries:

sudo apt-get purge "libnvinfer*"

• Install all libnvinfer libraries for your cuda version. Run apt-cache policy libnvinfer8 to check available versions.

For cuda-11.4 and libnvinfer 8.2.5.1:

sudo apt install libnvinfer8=8.2.5-1+cuda11.4 libnvinfer-plugin8=8.2.5-1+cuda11.4 libnvparsers8=8.2.5-1+cuda11.4 libnvonnxparsers8=8.2.5-1+cuda11.4 libnvinfer-dev=8.2.5-1+cuda11.4 libnvinfer-plugin-dev=8.2.5-1+cuda11.4 libnvparsers-dev=8.2.5-1+cuda11.4 libnvonnxparsers-dev=8.2.5-1+cuda11.4 cuda-cudart-dev-11-4 libcublas-dev-11-4

For cuda-11.6 and libinfer 8.4.3 (tested also with cuda-11.8):

sudo apt install libnvinfer8=8.4.3-1+cuda11.6 libnvinfer-plugin8=8.4.3-1+cuda11.6 libnvparsers8=8.4.3-1+cuda11.6 libnvonnxparsers8=8.4.3-1+cuda11.6 libnvinfer-dev=8.4.3-1+cuda11.6 libnvinfer-plugin-dev=8.4.3-1+cuda11.6 libnvinfer-bin=8.4.3-1+cuda11.6 libnvparsers-dev=8.4.3-1+cuda11.6 libnvonnxparsers-dev=8.4.3-1+cuda11.6 cuda-cudart-dev-11-6 libcublas-dev-11-6

Hope it helps.

First, onnxruntime python packages, “onnxruntime” and “onnxruntime-gpu”, follow manylinux2014(pep-0599 ) standard. But the gpu one, onnxruntime-gpu, isn’t fully compliant.

The PEP 599 policy says: “The wheel’s binary executables or shared objects may not link against externally-provided libraries except those in the following list”

• libgcc_s.so.1
• libstdc++.so.6
• libm.so.6
• libdl.so.2
• librt.so.1
• libc.so.6
• libnsl.so.1
• libutil.so.1
• libpthread.so.0
• libresolv.so.2
• libX11.so.6
• libXext.so.6
• libXrender.so.1
• libICE.so.6
• libSM.so.6
• libGL.so.1
• libgobject-2.0.so.0
• libgthread-2.0.so.0
• libglib-2.0.so.0

But we need CUDA. And CUDA isn’t in the list. BTW, if you run ldd with onnxruntime’s cpu only package, “onnxruntime”, you won’t see the error.

The policy was designed as any external dependency should be packed into the wheel file. However, we can’t. Because,

1. It will make the package over size. Pypi.org has a 100MB per wheel file size limit.
2. It could cause license issues. I’m not sure if we could redistribute Nvidia’s CUDA libraries.

So we did a dirty hack. Before pack the wheel, we patch the so file to pretend it doesn’t depend on CUDA. To cheat on manylinux’s auditwheel tool. Then we pack the wheel and manually load the CUDA libraries. The error message you saw is caused the tool for patching *.so files: patchelf. If we don’t use the tool, we won’t have this issue.

Alternatively, we could modify the policy. Patch the auditwheel tool, add a custom policy file to whitelist CUDA libraries. The file is: https://github.com/pypa/auditwheel/blob/main/src/auditwheel/policy/manylinux-policy.json . See #144 for more information.

(Hi @adk9, the above answer is only for onnxruntime inference packages. onnxruntime-training package is built in a special way that I’m not familiar. )

Stumbled across this issue. FWIW, in newer auditwheel there is an option to exclude shared objects that will be provided in a different manner. This is the PR that added the option --exclude pypa/auditwheel#368, specifically for the use case described here:

$ auditwheel repair --help
usage: auditwheel repair [-h] [--plat PLATFORM] [-L LIB_SDIR] [-w WHEEL_DIR] [--no-update-tags] [--strip] [--exclude EXCLUDE]
                         [--only-plat]
                         WHEEL_FILE [WHEEL_FILE ...]

Vendor in external shared library dependencies of a wheel.
If multiple wheels are specified, an error processing one
wheel will abort processing of subsequent wheels.
...
 --exclude EXCLUDE     Exclude SONAME from grafting into the resulting wheel (can be specified multiple times)

I’m also running into the same issue with onnxruntime-training (1.9.0) and onnxruntime-gpu (1.9.0) wheels installed from PyPI, trying to train a simple model using the CUDA EP. @snnn, the suggested fix above is not clear to me; can you elaborate on it?