tensorflow: TF-TRT Warning: Could not find TensorRT

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

2.15.0

Custom code

No

OS platform and distribution

Ubuntu 22.04

Mobile device

No response

Python version

3.11

Bazel version

No response

GCC/compiler version

11.2

CUDA/cuDNN version

12.4

GPU model and memory

RTX 3050 TI

Current behavior?

Hey everybody,

I have tried installing tensorflow from my ubuntu 22.04 terminal with anaconda environment. I am getting Can not find TensorRT. I have a 4 monitor setup with Ubuntu and the 550 NVIDIA driver for the RTX 3050 TI does not work on my machine. I have to run the 535 driver, and I think 12.4 CUDA is supported. It installed the 550 driver automatically and I revert back to the 535 driver. But after all that I still ended up with the error below.

I have tried uninstalling, reinstalling, and can’t get it to work. I am a graduate student taking a Machine Learning course. I am spending more time debugging this issue then actually coding.

Could somebody help me resolve?

Standalone code to reproduce the issue

+---------------------------------------------------------------------------------------+                                                                                                                          
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |                                                                                                                          
|-----------------------------------------+----------------------+----------------------+                                                                                                                          
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |                                                                                                                          
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |                                                                                                                          
|                                         |                      |               MIG M. |                                                                                                                          
|=========================================+======================+======================|                                                                                                                          
|   0  NVIDIA GeForce RTX 3050 ...    Off | 00000000:01:00.0  On |                  N/A |                                                                                                                          
| N/A   66C    P8               8W /  35W |   1011MiB /  4096MiB |      0%      Default |                                                                                                                          
|                                         |                      |                  N/A |                                                                                                                          
+-----------------------------------------+----------------------+----------------------+                                                                                                                          
                                                                                                                                                                                                                   
+---------------------------------------------------------------------------------------+                                                                                                                          
| Processes:                                                                            |                                                                                                                          
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |                                                                                                                          
|        ID   ID                                                             Usage      |                                                                                                                          
|=======================================================================================|                                                                                                                          
|    0   N/A  N/A      2812      G   /usr/lib/xorg/Xorg                          648MiB |                                                                                                                          
|    0   N/A  N/A      3023      G   /usr/bin/gnome-shell                        148MiB |                                                                                                                          
|    0   N/A  N/A     78082      G   ...az/anaconda3/envs/skynet/bin/python        1MiB |                                                                                                                          
|    0   N/A  N/A     78475      G   ...seed-version=20240329-165146.919000       96MiB |                                                                                                                          
|    0   N/A  N/A     79970      G   ...gnu/webkit2gtk-4.0/WebKitWebProcess       68MiB |                                                                                                                          
+---------------------------------------------------------------------------------------+   
(base) thenaz@Skynet:~/Downloads$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 2291 (rev a1)
(base) thenaz@Skynet:~/Downloads$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver                                                                                                                                                                              
Copyright (c) 2005-2024 NVIDIA Corporation                                                                                                                                                                         
Built on Tue_Feb_27_16:19:38_PST_2024                                                                                                                                                                              
Cuda compilation tools, release 12.4, V12.4.99                                                                                                                                                                     
Build cuda_12.4.r12.4/compiler.33961263_0

Relevant log output

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"                                                                                  
2024-03-30 14:29:58.668656: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.                                                                                                                                 
2024-03-30 14:29:58.691384: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 
2024-03-30 14:29:58.691405: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered  
2024-03-30 14:29:58.692006: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered                                                                                                                                                                                                                 
2024-03-30 14:29:58.695703: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.                         
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-30 14:29:59.119119: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-30 14:29:59.660586: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-30 14:29:59.675221: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-30 14:29:59.675340: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 8
  • Comments: 49

Most upvoted comments

i think pip install tensorflow[cuda] will solve the issue but i am not quite sure

Wanted to add more onto this, I spent a day trying various WSL Tensorflow installation methods and still no luck.

For context, I’m on Windows 11, with an i7-13700 and an RTX A4000 Series GPU.

Here are my printouts:

nvidia-smi
Fri May  3 09:50:02 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76.01              Driver Version: 552.22         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000               On  |   00000000:01:00.0 Off |                  Off |
| 45%   37C    P0             35W /  140W |       0MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
 which python
/home/bplaster/anaconda3/bin/python

interestingly, after using pip install tensorflow[and-cuda] within the WSL environment, I was getting this:

python3 -c "import tensorrt; print(tensorrt.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'tensorrt'

I attempted to resolve this with: pip install tensorrt

Collecting tensorrt
  Downloading tensorrt-10.0.1.tar.gz (16 kB)
  Preparing metadata (setup.py) ... done
Collecting tensorrt-cu12 (from tensorrt)
  Downloading tensorrt-cu12-10.0.1.tar.gz (18 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: tensorrt, tensorrt-cu12
  Building wheel for tensorrt (setup.py) ... done
  Created wheel for tensorrt: filename=tensorrt-10.0.1-py2.py3-none-any.whl size=16333 sha256=19d103fecc3bfb19836fe815d369e65171978f61bb476e89ae84b857953f64e7
  Stored in directory: /home/bplaster/.cache/pip/wheels/31/90/ef/53ad98d9a1bd660c0177aa1ea91bde288edfbe4d15621ca472
  Building wheel for tensorrt-cu12 (setup.py) ... done
  Created wheel for tensorrt-cu12: filename=tensorrt_cu12-10.0.1-py2.py3-none-any.whl size=17549 sha256=5877c62938e16384e9c969b00a94dea5bda1bea7de2db0249bcfd8520d5082fd
  Stored in directory: /home/bplaster/.cache/pip/wheels/c5/08/30/a058eaf14eeeabb3799e3abd3a2a534649123824dcd9695876
Successfully built tensorrt tensorrt-cu12
Installing collected packages: tensorrt-cu12, tensorrt
Successfully installed tensorrt-10.0.1 tensorrt-cu12-10.0.1

Checking the nvidia compatibility matrrix, this should be good to go: image

Now, we see that TensorRT is there when called:

python3 -c "import tensorrt; print(tensorrt.__file__)"
/home/bplaster/anaconda3/lib/python3.11/site-packages/tensorrt/__init__.py

Yet we still get the following message when checking the Tensorflow version:

python -c "import tensorflow as tf; print(tf.__version__)"
2024-05-03 09:51:40.606740: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-03 09:51:40.632018: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-03 09:51:41.004536: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2.16.1

Finally, confirming that the GPU cannot be seen by Tensorflow for some reason:

python -c "import tensorflow as tf; print(tf.config.list_physical_devices())"
2024-05-03 10:01:27.648034: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-03 10:01:27.673662: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-03 10:01:28.043090: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-03 10:01:30.488133: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-05-03 10:01:30.501591: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]

This is all done on a completely fresh WSL2 Ubuntu environment, all done in the (base) conda environment. Hope this is helpful.

I’m frustrated with tensorflow, 1 decade after launch and its still a nightmare to get it to work with gpu.

EDIT: hours later, I created fresh venv and installed tensorflow[and-cuda] alone, then installed tensorrt. The error in this gituhub issue vanished, but still no gpu. After that, I had to do path manipulation as indicated here and it worked.

Setting TF_CPP_MAX_VLOG_LEVEL=3 in my environment (WSL2 Ubuntu) gives the following specific error:

2024-06-04 15:42:11.291621: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/zilla/miniconda3/lib/:/home/zilla/miniconda3/lib/python3.12/site-packages/nvidia/cudnn/lib
2024-06-04 15:42:11.291675: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/zilla/miniconda3/lib/:/home/zilla/miniconda3/lib/python3.12/site-packages/nvidia/cudnn/lib
2024-06-04 15:42:11.291687: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

While I have TensorRT, and thus libnvinfer, installed, the actual libraries are named libnvinfer.so.10 and libnvinfer_plugin.so.10 Just as a test, I did the following:

~/miniconda3/lib$ ln -s python3.12/site-packages/tensorrt_libs/libnvinfer.so.10 libnvinfer.so.8.6.1
~/miniconda3/lib$ ln -s python3.12/site-packages/tensorrt_libs/libnvinfer_plugin.so.10 libnvinfer_plugin.so.8.6.1

And the warning about not finding TensorRT diapered. Of course, if the libraries are not binary compatible, this might lead to further issues down the line, but at least it hints at what the problem is.

I have same issue on python 3.8 image on docker

" tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT"

Hi, I was able to get it fixed. Here are the steps:

extract and move that folder to /usr/local/TensorRT-8.6.1

tar -xzvf TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz

sudo mv TensorRT-8.6.1.6 /usr/local/TensorRT-8.6.1
  • Update the path
code ~/.bashrc
export PATH=/usr/local/cuda-12.1/bin:/usr/local/TensorRT-8.6.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:/usr/local/TensorRT-8.6.1/lib:$LD_LIBRARY_PATH
source ~/.bashrc

*now go to the extracted folder and install tensorrt from there

cd /usr/local/TensorRT-8.6.1/python


pip install tensorrt-8.6.1-cp310-none-linux_x86_64.whl

pip install tensorrt_dispatch-8.6.1-cp310-none-linux_x86_64.whl

pip install tensorrt_lean-8.6.1-cp310-none-linux_x86_64.whl

Hope this helps, i was using python 3.10 tensor rt folder has lib for other python versions to

Couldn’t resolve TF-TRT Warning: Could not find TensorRT

I’m using wsl2 in windows 11.

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Previously, I wasn’t able to have GPU as the backend, I had tried all the methods in installing tensorflow in wsl.

But, trying with python3.11 and tensorflow 2.15.1, got me access to the GPU backend.

conda create -n tmpenv python=3.11
conda activate tmpenv
pip install tensorflow[and-cuda]==2.15.1

versions:

tensorflow                    2.15.1
tensorrt                      10.0.1

nvidia-smi:

Tue May 21 15:45:23 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76.01              Driver Version: 552.44         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4080 ...    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   38C    P4             17W /   55W |       0MiB /  12282MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
2024-05-21 15:46:26.487553: I external/local_xla/xla/service/service.cc:168] XLA service 0x7fcc3f7b2c00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-05-21 15:46:26.487663: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 4080 Laptop GPU, Compute Capability 8.9

I’ve tried all ways of installing tensorrt - pip, .deb, .tar; updated path variable, etc. Its importing in python shell

>>> import tensorrt as trt
>>> trt.__version__
'10.0.1'
>>> trt.__file__
'/home/mohan/miniconda3/envs/tmpenv/lib/python3.11/site-packages/tensorrt/__init__.py'

All the files are available (.tar installation method)

(tmpenv) mohan@LAPTOP-INPO8147:~$ ls TensorRT-10.0.1.6/
bin  data  doc  include  lib  onnx_graphsurgeon  python  samples  targets

(tmpenv) mohan@LAPTOP-INPO8147:~$ ls TensorRT-10.0.1.6/lib/
libnvinfer.so                          libnvinfer_plugin.so.10.0.1
libnvinfer.so.10                       libnvinfer_plugin_static.a
libnvinfer.so.10.0.1                   libnvinfer_static.a
libnvinfer_builder_resource.so.10.0.1  libnvinfer_vc_plugin.so
libnvinfer_dispatch.so                 libnvinfer_vc_plugin.so.10
libnvinfer_dispatch.so.10              libnvinfer_vc_plugin.so.10.0.1
libnvinfer_dispatch.so.10.0.1          libnvinfer_vc_plugin_static.a
libnvinfer_dispatch_static.a           libnvonnxparser.so
libnvinfer_lean.so                     libnvonnxparser.so.10
libnvinfer_lean.so.10                  libnvonnxparser.so.10.0.1
libnvinfer_lean.so.10.0.1              libnvonnxparser_static.a
libnvinfer_lean_static.a               libonnx_proto.a
libnvinfer_plugin.so                   stubs
libnvinfer_plugin.so.10

(tmpenv) mohan@LAPTOP-INPO8147:~/TensorRT-10.0.1.6/python$ ls
tensorrt-10.0.1-cp310-none-linux_x86_64.whl
tensorrt-10.0.1-cp311-none-linux_x86_64.whl
tensorrt-10.0.1-cp312-none-linux_x86_64.whl
tensorrt-10.0.1-cp38-none-linux_x86_64.whl
tensorrt-10.0.1-cp39-none-linux_x86_64.whl
tensorrt_dispatch-10.0.1-cp310-none-linux_x86_64.whl
tensorrt_dispatch-10.0.1-cp311-none-linux_x86_64.whl
tensorrt_dispatch-10.0.1-cp312-none-linux_x86_64.whl
tensorrt_dispatch-10.0.1-cp38-none-linux_x86_64.whl
tensorrt_dispatch-10.0.1-cp39-none-linux_x86_64.whl
tensorrt_lean-10.0.1-cp310-none-linux_x86_64.whl
tensorrt_lean-10.0.1-cp311-none-linux_x86_64.whl
tensorrt_lean-10.0.1-cp312-none-linux_x86_64.whl
tensorrt_lean-10.0.1-cp38-none-linux_x86_64.whl
tensorrt_lean-10.0.1-cp39-none-linux_x86_64.whl

I’ve also used the files from .tar method to install using pip:

(tmpenv) mohan@LAPTOP-INPO8147:~/TensorRT-10.0.1.6/python$ python3 -m pip install tensorrt-10.0.1-cp311-none-linux_x86_64.whl
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing ./tensorrt-10.0.1-cp311-none-linux_x86_64.whl
tensorrt is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

(tmpenv) mohan@LAPTOP-INPO8147:~/TensorRT-10.0.1.6/python$ python3 -m pip install tensorrt_dispatch-10.0.1-cp311-none-linux_x86_64.whl
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing ./tensorrt_dispatch-10.0.1-cp311-none-linux_x86_64.whl
tensorrt-dispatch is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

(tmpenv) mohan@LAPTOP-INPO8147:~/TensorRT-10.0.1.6/python$ python3 -m pip install tensorrt_lean-10.0.1-cp311-none-linux_x86_64.whl
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing ./tensorrt_lean-10.0.1-cp311-none-linux_x86_64.whl
tensorrt-lean is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

But still I get this error

2024-05-21 15:46:19.659564: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

Got exactly into the same problem with Python 3.11, CUDA 12.4 and TensorRT 10.1. Tried suggestion by @Zilla - it appears it’s a working solution / workaround.

What I did:

export TF_CPP_MAX_VLOG_LEVEL=3
# Ran my Python program and got:
2024-06-17 21:54:19.501338: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-12.4/lib64:/usr/local/lib/python3.11/dist-packages/tensorrt/lib:/usr/local/cuda-12.4/lib64:/usr/local/lib/python3.11/dist-packages/tensorrt/lib:
2024-06-17 21:54:19.501403: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-12.4/lib64:/usr/local/lib/python3.11/dist-packages/tensorrt/lib:/usr/local/cuda-12.4/lib64:/usr/local/lib/python3.11/dist-packages/tensorrt/lib:

Then I did:

cd /usr/local/lib/python3.11/dist-packages/tensorrt/lib
sudo ln -s libnvinfer.so.10 libnvinfer.so.8.6.1
sudo ln -s libnvinfer_plugin.so.10 libnvinfer_plugin.so.8.6.1

My application is not complaining about missing TensorRT library.

Setting TF_CPP_MAX_VLOG_LEVEL=3 in my environment (WSL2 Ubuntu) gives the following specific error:

2024-06-04 15:42:11.291621: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/zilla/miniconda3/lib/:/home/zilla/miniconda3/lib/python3.12/site-packages/nvidia/cudnn/lib
2024-06-04 15:42:11.291675: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/zilla/miniconda3/lib/:/home/zilla/miniconda3/lib/python3.12/site-packages/nvidia/cudnn/lib
2024-06-04 15:42:11.291687: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

While I have TensorRT, and thus libnvinfer, installed, the actual libraries are named libnvinfer.so.10 and libnvinfer_plugin.so.10 Just as a test, I did the following:

~/miniconda3/lib$ ln -s python3.12/site-packages/tensorrt_libs/libnvinfer.so.10 libnvinfer.so.8.6.1
~/miniconda3/lib$ ln -s python3.12/site-packages/tensorrt_libs/libnvinfer_plugin.so.10 libnvinfer_plugin.so.8.6.1

And the warning about not finding TensorRT diapered. Of course, if the libraries are not binary compatible, this might lead to further issues down the line, but at least it hints at what the problem is.

Yes. It works. I have the miniconda env “cuda12”, cuda 12, tensorrt 10 on Ubuntu 22, WSL2.

That is my commands:

cd ~/miniconda3/envs/cuda12/lib/python3.11/site-packages/tensorrt_libs/
ln -s libnvinfer.so.10 libnvinfer.so.8.6.1
ln -s libnvinfer_plugin.so.10 libnvinfer_plugin.so.8.6.1

I think that the only way is to find the compatible versions for python, tensorflow, nvidia driver, cuda, cudnn and tensorrt. I’ll try the tested build configurations from here in WSL and Ubuntu. Hope it works 🙏

So, after trying out pretty much everything else, I can confirm that this worked out and my GPU is now also detected. I am on Ubuntu 22.0.4.4 running Tensorflow 2.16.

Many thanks @BigKhalPablo, saved my day 😉

Cheers, Eric

Same issue:

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"    
2024-05-08 12:14:54.731695: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-08 12:14:55.226452: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-08 12:14:55.882997: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:2b:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-05-08 12:14:55.911562: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:2b:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-05-08 12:14:55.911640: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:2b:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

My nvidia drivers and cuda version:

nvidia-smi
Wed May  8 12:06:07 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76.01              Driver Version: 552.22         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 Ti     On  |   00000000:2B:00.0  On |                  N/A |
| 30%   31C    P0             42W /  200W |     979MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

I’ve installed tensorflow and tensorrt through pip:

pip install tensorflow[and-cuda]
pip install tensorrt

I have follow the guide: Install TensorFlow with pip and added path as in Install Tensorflow & Pytorch with CUDA [Linux | WSL2] but I’m still getting the same warning (TF-TRT Warning: Could not find TensorRT).

Does any anyone managed to find a solution?

Just commenting to say I am experiencing the exact same issue. Some follow up information.

which python3
>>> /home/bplaster/anaconda3/bin/python3
python3 -c "import tensorrt; print(tensorrt.__file__)"
>>> /home/bplaster/anaconda3/lib/python3.11/site-packages/tensorrt/__init__.py
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices())"
>>> 2024-05-02 15:35:07.243819: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-02 15:35:07.269922: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-02 15:35:07.628229: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-02 15:35:08.153905: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-05-02 15:35:08.156012: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
 nvidia-smi
Thu May  2 15:46:11 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.76.01              Driver Version: 552.22         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000               On  |   00000000:01:00.0 Off |                  Off |
| 41%   37C    P8              6W /  140W |      10MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Hi @MachinesOnTheRise ,

You might need to install compatible TensorRT version from Nvidia. Since this is a optimization library for Inferences and developed by Nvidia for their specific GPUs, it may not come bundled with TF package. You can use tensorflow[and-cuda] package for installing TF with required cuda/cudnn packages as below.But they seems don’t have tensorrt bundled.

https://github.com/tensorflow/tensorflow/blob/ced28148be8cec1f234886f21c7e528c374032b8/tensorflow/tools/pip_package/setup.py#L151-L166

Thank you all for your help. I was able to resolve it by adding the tensorrtlibs to LD_LIBRARY_PATH. image

W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT I’m getting this error while running dreambooth lora on colab . pip install tensorflow[and-cuda] – This also not solved the problem. Any solutions ?

I got exactly the same question like you, any ideas?

I think that the only way is to find the compatible versions for python, tensorflow, nvidia driver, cuda, cudnn and tensorrt. I’ll try the tested build configurations from here in WSL and Ubuntu. Hope it works 🙏