tensorflow: TF-TRT Warning: Could not find TensorRT
Issue type
Build/Install
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
tf2.12, tf2.13
Custom code
Yes
OS platform and distribution
Linux Ubuntu 20.04
Mobile device
No response
Python version
3.10, 3.11
Bazel version
No response
GCC/compiler version
9.40
CUDA/cuDNN version
CUDA 11.8, cuDNN8.6
GPU model and memory
RTX2060
Current behavior?
import tensorflow as tf2023-08-03 17:42:07.337886: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-08-03 17:42:07.926267: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Standalone code to reproduce the issue
$ conda create --name tf python=3.10
$ conda activate tf
$ conda install -c conda-forge cudatoolkit=11.8.0
$ pip install nvidia-cudnn-cu11==8.6.0.163
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/:$CUDNN_PATH/lib:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))
Relevant log output
I have already installed cudatoolkit 11.8 written in the above commands. But It shows CUDA Toolkit is not installed while giving the command.
$ nvcc --version
CUDA Toolkit is not installed.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 61
Hi @mikechen66 ,
Please follow the correct configuration and follow the documentation
Please also have a look of similar issue at https://github.com/tensorflow/tensorflow/issues/41060.
Thanks!
Try adding this command
!pip install tensorflow-gpu==2.8.0Hi @mikechen66 ,
I am assuming you are using prebuilt binaries from Pypi.If you are building from source please confirm.
Could you please confirm whether CUDNN PATH setting done like below. Refer step 4 in attached documentation source for more details on configuring GPU.
Also please run the command
nvidia-smito check whether Nvidia driver installed or not.Please ignore TensorRT warning as this is optional and it won’t affect GPU. Thanks!
@skonto you’re right, 2.15-post1 is looking for
libnvinfer_plugin.so.8.6.1and libnvinfer.so.8.6.1You can know which file your tf is finding by using
strace -e open,openat python -c "import tensorflow as tf"in your venv.I recall that in before, tf2.10 and below looks for libnvinfer.so.7 while the tensorrt python prebuild package only offers libnvinfer.so.8, so you do stupid things
ln -s libnvinfer.so.8 libnvinfer.so.7to get it working,also the file used to be in tensorrt folder, but now, the tensorrt package is not included in
pip install tensorflow[and-cuda](and-cuda package). Thus you need to install tensorrt, too. The tensorrt package on pypi comes with 8.6.1. However, the package only gives libnvinfer.so.8 where tensorflow is looking for libnvinfer.8.6.1.Thus the way to solve this is to go to your venv site-packages folder, find tensorrt_libs folder,
(in my case tf version 2.15-post1)
ln -s libnvinfer_plugin.so.8 libnvinfer_plugin.so.8.6.1ln -s libnvinfer.so.8 libnvinfer.so.8.6.1and make sure tensorrt_libs folder is in the LD_LIBRARY_PATH
(maybe
TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0offerslibnvinfer.so.8.6.1, but it works for me though)It’s quite difficult to manage everything with all those changes during time, but a logical assumption is that
tensorflow[and-cuda]will remain to be a preferable setup in the future. So, for the most recent tensorflow and CUDA:pip install tensorflow[and-cuda]==v2.16.0-rc0(8, 6, 1)
Download related tar.gz https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz , extract it and set its
libto the environment variable LD_LIBRARY_PATH:@zeke-john
Why not try this:
strace -e open,openat python -c "import tensorflow as tf" 2>&1 | grep "libnvinfer\|TF-TRT"This would tell you what file tensorflow is looking for, and just find the file either from the targz package or tensorrt package on pypi, then add the folder into your LD_LIBRARY_PATH and softlink the file if necessary.Thank you for mentioning this technique, I am using TensorFlow 2.16.1(latest) and faced the same problem. When I used the above tool, I found that libcudnn.so.8 was missing from tensorrt_libs. Hence, installed CUDNN. Also soft linked the so.8 files to 8.6.1. Put CUDA lib path, CUDNN lib path and tensorrt_libs location in LD_LIBRARY_PATH environment variable. It began to work after this.
I am attaching TensorRT Tensorflow compatibility matrix hoping it might be useful;
Actually I think Cuda and TensorRT are not interdependent. I have used cuda 12.0 (even 12.4) with TensorRT 8.6.1.
I was using Tensorflow 2.16.1. Tensorflow may complain about missing libraries. For this, soft linking of files is required to be done in cuda lib folder. like this:
ln -s <required_file>.so.12 <req_file>.so.11Where the required files are: libcublaslt, libcudart, libcufft, libcusparse The missing libraries maybe shown before the tensorrt warning. Without resolving them, we cannot use GPU itself. The Cuda lib folder must be put in LD_LIBRARY_PATH.
Then I installed cudnn 8.9.6. Put CUDNN lib in LD_LIBRARY_PATH.
I installed tensorrt from pypi.
pip install tensorrtThis should install 8.6.1 (latest General Access). otherwise specify the version too. Then I installed tensorrt_libs and tensorrt_bindings (both version 8.6.1) in the same way. In the tensorrt_libs folder .so.8 files are present. They need to be soft linked to 8.6.1 like this.ln -s <required_file>.so.8 <req_file>.so.8.6.1where required files are: libnvinfer, libnvinfer_plugin.
tensorrt_libs should also be put in LD_LIBRARY_PATH environment variable. Overall this is what I specified in LD_LIBRARY_PATH variable after doing all the above steps:
export LD_LIBRARY_PATH=/usr/local/cudnn-linux-x86_64-8.9.6.50_cuda12-archive/lib:/home/hegdetej/test/lib/python3.10/site-packages/tensorrt_libs/:/usr/local/cuda-12.0/lib64/PS:- Even I tried using TensorRT apt package, but it didn’t seem to work for some reason, hence had to use this lengthy workaround.
it’s worked thanks
I’m genuinely impressed by TF’s ability to never work with GPUs for me. I’ve tried every single suggestion in this thread and many others, on several different machines, both with and without virtualenvs and at no point can I get tensorflow to see TensorRT and therefore my GPU.
JAX can see my GPU just fine.
Edit: I have finally been able to get tensorflow to see TensorRT, but despite NO LIBRARY ERRORS it still says there are some:
“
the missing libraries mentioned above” where above? These are all the output lines.For aws sagemaker this fixed it for me but i had to close the terminal and open a new one:
pip install tensorrtI would like to thank ipaleka, KardRi , TXH2020 and other contributors. After installingTensortRT 8.6.1.6, I solve the issue of “TF-TRT Warning: Could not find TensorRT”.
Installing as same as the above-mentioned except TensortRT 8.6.1.6
Exporting
print tf version
2.16.1
print tensorrt version
8.6.1
However, after running the following script I have a new issue Successful NUMA node read from SysFS had negative value (-1).
…Successful NUMA node read from SysFS had negative value (-1),… [PhysicalDevice(name=‘/physical_device:GPU:0’, device_type=‘GPU’)
There are temporary solutions as follows.
https://stackoverflow.com/questions/44232898/memoryerror-in-tensorflow-and-successful-numa-node-read-from-sysfs-had-negativ
https://gist.github.com/zrruziev/b93e1292bf2ee39284f834ec7397ee9f
Ubuntu 22.04 with
Driver Version: 535.161.08 CUDA Version: 12.2Fixed by https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar I install withwget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz, andexport LD_LIBRARY_PATH=/data/TensorRT-8.6.1.6/lib:$LD_LIBRARY_PATH(remind replace TensorRT Path for you)Where do I need to copy these 2 files to? for ex. TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0 fyii. I am on Ubuntu 22.04 on WSL2 on Windows 11
The following snippet resolved the tensorrt problem completely on my machine.
Thank ipaleka for the tar installation method and thank benz0li for instructions.
I understand tar is a much more flexible installation method. However, I still find out a strange issue during TensorRT tar installation. If executing the single command “import tensorrt”, it is no problem. However, it shows error while executing “import tensorflow as tf”, it shows TF-TRT warning: Could not find TensorRT.
Part One. Installation
1.Download TensorRT
https://developer.nvidia.com/tensorrt/download/10x 10.0.1.6 GA(General Availability)
2.Unzip TensorRT to tf_tensorrt fileholder
3.List Subdirectories
bin data doc include lib onnx_graphsurgeon python samples targets
4.Add TensorRT lib directory to the environment variable LD_LIBRARY_PATH:
# Update
# Go back to tf env
5.Install python TensorRT wheel file
# Shorten tensorrt version to v10.0.1 from v10.0.1.6
6.Install Python onnx-graphsurgeon wheel
Part Two. Can find tensorrt in the conda list but show error with TensorFlow
1.Check the installation
tensorrt 10.0.0.1 pypi_0 pypi
2.Check tensorrt (it seems good)
It shows the TensorRT version correctly.
10.0.1
3. Check tf version
2024-05-03 22:34:03.502201: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2.16.1
4. Execute the command
2024-05-03 22:35:39.353689: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
True
Please indicate where is wrong?
Thanks in advance.
Starting from TF2.12v
tensorflow-gpubecame redundant and installingtensorflowautomatically installs GPU version also.