tensorflow: TF-TRT Warning: Could not find TensorRT

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

source

TensorFlow version

tf2.12, tf2.13

Custom code

Yes

OS platform and distribution

Linux Ubuntu 20.04

Mobile device

No response

Python version

3.10, 3.11

Bazel version

No response

GCC/compiler version

9.40

CUDA/cuDNN version

CUDA 11.8, cuDNN8.6

GPU model and memory

RTX2060

Current behavior?

import tensorflow as tf2023-08-03 17:42:07.337886: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-08-03 17:42:07.926267: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Standalone code to reproduce the issue

$ conda create --name tf python=3.10
$ conda activate tf

$ conda install -c conda-forge cudatoolkit=11.8.0
$ pip install nvidia-cudnn-cu11==8.6.0.163


mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/:$CUDNN_PATH/lib:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))

Relevant log output

I have already installed cudatoolkit 11.8 written in the above commands. But It shows CUDA Toolkit is not installed while giving the command. 

$ nvcc --version
CUDA Toolkit is not installed.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 61

Most upvoted comments

Hi @mikechen66 ,

Please follow the correct configuration and follow the documentation

Please also have a look of similar issue at https://github.com/tensorflow/tensorflow/issues/41060.

Thanks!

Try adding this command

!pip install tensorflow-gpu==2.8.0

Hi @mikechen66 ,

I am assuming you are using prebuilt binaries from Pypi.If you are building from source please confirm.

Could you please confirm whether CUDNN PATH setting done like below. Refer step 4 in attached documentation source for more details on configuring GPU.

CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH

Also please run the command nvidia-smi to check whether Nvidia driver installed or not.

Please ignore TensorRT warning as this is optional and it won’t affect GPU. Thanks!

@skonto you’re right, 2.15-post1 is looking for libnvinfer_plugin.so.8.6.1 and libnvinfer.so.8.6.1

absl::StatusOr<void*> GetNvInferDsoHandle() {
#if defined(PLATFORM_WINDOWS)
  return GetDsoHandle("nvinfer", "");
#else
  return GetDsoHandle("nvinfer", GetTensorRTVersion());
#endif
}

absl::StatusOr<void*> GetNvInferPluginDsoHandle() {
#if defined(PLATFORM_WINDOWS)
  return GetDsoHandle("nvinfer_plugin", "");
#else
  return GetDsoHandle("nvinfer_plugin", GetTensorRTVersion());
#endif
}

You can know which file your tf is finding by using strace -e open,openat python -c "import tensorflow as tf" in your venv.

I recall that in before, tf2.10 and below looks for libnvinfer.so.7 while the tensorrt python prebuild package only offers libnvinfer.so.8, so you do stupid things ln -s libnvinfer.so.8 libnvinfer.so.7 to get it working,

also the file used to be in tensorrt folder, but now, the tensorrt package is not included in pip install tensorflow[and-cuda] (and-cuda package). Thus you need to install tensorrt, too. The tensorrt package on pypi comes with 8.6.1. However, the package only gives libnvinfer.so.8 where tensorflow is looking for libnvinfer.8.6.1.

Thus the way to solve this is to go to your venv site-packages folder, find tensorrt_libs folder,

(in my case tf version 2.15-post1)

ln -s libnvinfer_plugin.so.8 libnvinfer_plugin.so.8.6.1

ln -s libnvinfer.so.8 libnvinfer.so.8.6.1

and make sure tensorrt_libs folder is in the LD_LIBRARY_PATH

(maybe TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0 offers libnvinfer.so.8.6.1, but it works for me though)

It’s quite difficult to manage everything with all those changes during time, but a logical assumption is that tensorflow[and-cuda] will remain to be a preferable setup in the future. So, for the most recent tensorflow and CUDA:

pip install tensorflow[and-cuda]==v2.16.0-rc0

python3 -c "import tensorflow.compiler as tf_cc; \
print(tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version())"

(8, 6, 1)

Download related tar.gz https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz , extract it and set its lib to the environment variable LD_LIBRARY_PATH:

export CUDNN_PATH=$HOME/venv/lib/python3.11/site-packages/nvidia/cudnn
export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$HOME/repos/TensorRT-8.6.1.6/lib:$LD_LIBRARY_PATH
export TF_ENABLE_ONEDNN_OPTS=0

@zeke-john

Why not try this: strace -e open,openat python -c "import tensorflow as tf" 2>&1 | grep "libnvinfer\|TF-TRT" This would tell you what file tensorflow is looking for, and just find the file either from the targz package or tensorrt package on pypi, then add the folder into your LD_LIBRARY_PATH and softlink the file if necessary.

@zeke-john

Why not try this: strace -e open,openat python -c "import tensorflow as tf" 2>&1 | grep "libnvinfer\|TF-TRT" This would tell you what file tensorflow is looking for, and just find the file either from the targz package or tensorrt package on pypi, then add the folder into your LD_LIBRARY_PATH and softlink the file if necessary.

Thank you for mentioning this technique, I am using TensorFlow 2.16.1(latest) and faced the same problem. When I used the above tool, I found that libcudnn.so.8 was missing from tensorrt_libs. Hence, installed CUDNN. Also soft linked the so.8 files to 8.6.1. Put CUDA lib path, CUDNN lib path and tensorrt_libs location in LD_LIBRARY_PATH environment variable. It began to work after this.

I am attaching TensorRT Tensorflow compatibility matrix hoping it might be useful;

TensorFlow version TensorRT version
1.7 3.0.4
1.11 4
1.13 5
2.1 6
2.10 7.2.2
2.16.1(latest) 8.6.1(latest stable)

Actually I think Cuda and TensorRT are not interdependent. I have used cuda 12.0 (even 12.4) with TensorRT 8.6.1.

I was using Tensorflow 2.16.1. Tensorflow may complain about missing libraries. For this, soft linking of files is required to be done in cuda lib folder. like this:

ln -s <required_file>.so.12 <req_file>.so.11

Where the required files are: libcublaslt, libcudart, libcufft, libcusparse The missing libraries maybe shown before the tensorrt warning. Without resolving them, we cannot use GPU itself. The Cuda lib folder must be put in LD_LIBRARY_PATH.

Then I installed cudnn 8.9.6. Put CUDNN lib in LD_LIBRARY_PATH.

I installed tensorrt from pypi. pip install tensorrt This should install 8.6.1 (latest General Access). otherwise specify the version too. Then I installed tensorrt_libs and tensorrt_bindings (both version 8.6.1) in the same way. In the tensorrt_libs folder .so.8 files are present. They need to be soft linked to 8.6.1 like this. ln -s <required_file>.so.8 <req_file>.so.8.6.1

where required files are: libnvinfer, libnvinfer_plugin.

tensorrt_libs should also be put in LD_LIBRARY_PATH environment variable. Overall this is what I specified in LD_LIBRARY_PATH variable after doing all the above steps:

export LD_LIBRARY_PATH=/usr/local/cudnn-linux-x86_64-8.9.6.50_cuda12-archive/lib:/home/hegdetej/test/lib/python3.10/site-packages/tensorrt_libs/:/usr/local/cuda-12.0/lib64/

PS:- Even I tried using TensorRT apt package, but it didn’t seem to work for some reason, hence had to use this lengthy workaround.

Try adding this command

!pip install tensorflow-gpu==2.8.0

it’s worked thanks

I’m genuinely impressed by TF’s ability to never work with GPUs for me. I’ve tried every single suggestion in this thread and many others, on several different machines, both with and without virtualenvs and at no point can I get tensorflow to see TensorRT and therefore my GPU.

JAX can see my GPU just fine.

Edit: I have finally been able to get tensorflow to see TensorRT, but despite NO LIBRARY ERRORS it still says there are some:

2024-06-01 22:21:10.657940: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-01 22:21:13.045312: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

“the missing libraries mentioned above” where above? These are all the output lines.

For aws sagemaker this fixed it for me but i had to close the terminal and open a new one: pip install tensorrt

I would like to thank ipaleka, KardRi , TXH2020 and other contributors. After installingTensortRT 8.6.1.6, I solve the issue of “TF-TRT Warning: Could not find TensorRT”.

Installing as same as the above-mentioned except TensortRT 8.6.1.6

Exporting

export LD_LIBRARY_PATH=/home/user/tf_tensorrt/TensorRT-8.6.1.6/lib:$LD_LIBRARY_PATH

print tf version

import tensorflow as tf 
print(tf.__version__)

2.16.1

print tensorrt version

import tensorrt as trt
print(trt.__version__)

8.6.1

However, after running the following script I have a new issue Successful NUMA node read from SysFS had negative value (-1).

import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))

…Successful NUMA node read from SysFS had negative value (-1),… [PhysicalDevice(name=‘/physical_device:GPU:0’, device_type=‘GPU’)

There are temporary solutions as follows.

https://stackoverflow.com/questions/44232898/memoryerror-in-tensorflow-and-successful-numa-node-read-from-sysfs-had-negativ

https://gist.github.com/zrruziev/b93e1292bf2ee39284f834ec7397ee9f

Ubuntu 22.04 with Driver Version: 535.161.08 CUDA Version: 12.2 Fixed by https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar I install with wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz, and export LD_LIBRARY_PATH=/data/TensorRT-8.6.1.6/lib:$LD_LIBRARY_PATH (remind replace TensorRT Path for you)

Where do I need to copy these 2 files to? for ex. TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0 fyii. I am on Ubuntu 22.04 on WSL2 on Windows 11

The following snippet resolved the tensorrt problem completely on my machine.

mkdir -p $CONDA_PREFIX/etc/conda/activate.d

echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'TENSORRT_PATH=$(dirname $(python -c "import tensorrt;print(tensorrt.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/:$CUDNN_PATH/lib:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

Thank ipaleka for the tar installation method and thank benz0li for instructions.

I understand tar is a much more flexible installation method. However, I still find out a strange issue during TensorRT tar installation. If executing the single command “import tensorrt”, it is no problem. However, it shows error while executing “import tensorflow as tf”, it shows TF-TRT warning: Could not find TensorRT.

Part One. Installation

1.Download TensorRT

https://developer.nvidia.com/tensorrt/download/10x 10.0.1.6 GA(General Availability)

2.Unzip TensorRT to tf_tensorrt fileholder

$ cd tf_tensorrt
$ tar -xzvf TensorRT-10.0.1.6.Linux.x86_64-gnu.cuda-12.4.tar.gz

3.List Subdirectories

$ ls TensorRT-10.0.1.6

bin data doc include lib onnx_graphsurgeon python samples targets

4.Add TensorRT lib directory to the environment variable LD_LIBRARY_PATH:

$ gedit ~/.bashrc
$ export LD_LIBRARY_PATH=/home/user/tf_tensorrt/TensorRT-10.0.1.6/lib:$LD_LIBRARY_PATH

# Update

$ source ~/.bashrc

# Go back to tf env

$ conda activate tf

5.Install python TensorRT wheel file

cd /home/user/tf_tensorrt/TensorRT-10.0.1.6/python

# Shorten tensorrt version to v10.0.1 from v10.0.1.6

$ pip install tensorrt-10.0.1-cp310-none-linux_x86_64.whl

6.Install Python onnx-graphsurgeon wheel

$ cd /home/user/tf_tensorrt/TensorRT-10.0.1.6/onnx_graphsurgeon
$ pip install onnx_graphsurgeon-0.5.0-py2.py3-none-any.whl

Part Two. Can find tensorrt in the conda list but show error with TensorFlow

1.Check the installation

$ conda list | grep tensorrt

tensorrt 10.0.0.1 pypi_0 pypi

2.Check tensorrt (it seems good)

It shows the TensorRT version correctly.

$ python
import tensorrt as trt
print(trt.__version__)

10.0.1

3. Check tf version

$ python -c "import tensorflow as tf; print(tf.__version__)"

2024-05-03 22:34:03.502201: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2.16.1

4. Execute the command

$ Python
import tensorrt as trt
import tensorflow as tf

2024-05-03 22:35:39.353689: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

print(tf.test.is_built_with_cuda())

True

print(tf.config.list_physical_devices('GPU'))`
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Please indicate where is wrong?

Thanks in advance.

Starting from TF2.12v tensorflow-gpu became redundant and installing tensorflow automatically installs GPU version also.