tensorflow: Could not load dynamic library 'libnvinfer_plugin.so.6'

Almost exactly the same issue with fresh install of Ubuntu.

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04 LTS
• TensorFlow installed from (source or binary): wheel
• TensorFlow version: 2.1.0
• Python version: 3.6.9
• Installed using virtualenv? pip? conda?: pip in virtualenv
• CUDA/cuDNN version: 10.0
• GPU model and memory: 1080 Ti

I followed this guide almost to the letter but ran into issues with nvidia-drm, which I used tty3 to work around as suggested by one of the comments.

When I run tf.test.is_gpu_available()…

2020-01-20 23:18:14.446621: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2020-01-20 23:18:14.448592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.92GiB deviceMemoryBandwidth: 451.17GiB/s

2020-01-20 23:18:14.449086: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64

2020-01-20 23:18:14.449397: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64
2020-01-20 23:18:14.449635: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64

2020-01-20 23:18:14.449877: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64

2020-01-20 23:18:14.450111: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64

2020-01-20 23:18:14.450330: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64

2020-01-20 23:18:14.450391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7

2020-01-20 23:18:14.450418: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

2020-01-20 23:18:14.450481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:

2020-01-20 23:18:14.450533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 

2020-01-20 23:18:14.450580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 

False

When I encountered this issue, I was able to get tf to detect my card with pip install tensorflow-gpu==2.0.

I got the same error as well. When I downgraded to TensorFlow 2.0.0 (with pip install tensorflow-gpu==2.0.0) it solved the issue.

The solution isn’t to downgrade tensorflow (there is no problem with tensorflow 2.1, just the install docs). Install the missing library:

apt install libnvinfer-plugin6=6.0.1-1+cuda10.1

There is a PR to fix the docs.

Just to clarify, the install instructions for Ubuntu 16.04 should be updated; there are several errors, one being that cuda-10-1 is the appropriate cuda package, the other is that libnvinfer-plugin6=6.0.1-1+cuda10.1 is missing.

# Add NVIDIA package repositories
# Add HTTPS support for apt-key
sudo apt-get install gnupg-curl
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.1.243-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_10.1.243-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo apt-get update
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt-get update

# Install NVIDIA driver
# Issue with driver install requires creating /usr/lib/nvidia
sudo mkdir /usr/lib/nvidia
sudo apt-get install --no-install-recommends nvidia-418
# Reboot. Check that GPUs are visible using the command: nvidia-smi

# Install development and runtime libraries (~4GB)
sudo apt-get install --no-install-recommends \
    cuda-10-1 \
    libcudnn7=7.6.4.38-1+cuda10.1  \
    libcudnn7-dev=7.6.4.38-1+cuda10.1


# Install TensorRT. Requires that libcudnn7 is installed above.
sudo apt-get install -y --no-install-recommends libnvinfer6=6.0.1-1+cuda10.1 \
    libnvinfer-dev=6.0.1-1+cuda10.1 \
    libnvinfer-plugin6=6.0.1-1+cuda10.1

Ok, figured mine out. Here’s an outline of what i did. I was running multiple TF versions, so I wanted to go the manual route. I’ve had bad experiences with NVIDIA and Ubuntu nixing each others packages.

checked the version of tensorflow I had installed by starting the venv and then

pip list | grep tensorflow tensorflow-gpu 2.1.0 tensorflow-estimator 2.1.0

Checked https://www.tensorflow.org/install/source#tested_build_configurations for the versions required.

Downloaded local installer of CUDA 10.1 from NVIDIA and ran the installer. Carefully navigated the menus so that it only copied the files to /usr/local/cuda-10.1 directory (ie: if it wanted to “help” me, i told it “no thanks”). Downloaded cuDNN 7.6.5.32.tar.gz and tar xvf’d it to my download directory, it created a “cuda” directory backed up /usr/local/cuda-10.1 by copying it Copied the cuda/include/cudnn.h to /usr/local/cuda-10.1/include Copied the contents of cuda/lib to /usr/local/cuda-10.1 Downloaded TensorRT 6 tarball Extracted TensorRT 6 to my download directory. Opened my .venv/bin/activate script

Added /usr/local/cuda-10.1 and TensorRT 6 to the LD_LIBRARY_PATH and EXPORT

LD_LIBRARY_PATH="/usr/local/cuda-10.1/lib64:/home/duane/Downloads/TensorRT-6.0.1.5/lib"
export LD_LIBRARY_PATH

then sourced the environment

. activate

python

import tensorflow as tf
tf.test.is_gpu_available()

Wah lah…

2020-02-28 19:39:35.895538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/device:GPU:0 with 7386 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:08:00.0, compute capability: 7.5)

Then I went into my living room, sat on the lounge, drank a bottle of vodka, and passed out.

Now I feel much better 😃

The solution isn’t to downgrade tensorflow (there is no problem with tensorflow 2.1, just the install docs). Install the missing library:

apt install libnvinfer-plugin6=6.0.1-1+cuda10.1

There is a PR to fix the docs.

It leads to another error

Reading package lists… Done Building dependency tree Reading state information… Done E: Unable to locate package libnvinfer-plugin6

This also arise when tensorflow-gpu is not installed, which is surprising

@ai-starter basically you need to upgrade to cuda 10.1 and install the correct version CUDNN and TensorRT libraries

You should be able to follow https://github.com/tensorflow/tensorflow/issues/35968#issuecomment-577329399

If it doesn’t work this is what worked for me - I suspect what works may depend on what you currently have installed / how you installed it.

# install nvidia key
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo apt-key add 7fa2af80.pub

# add repo's
sudo bash -c 'echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" > /etc/apt/sources.list.d/cuda.list'
sudo bash -c 'echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/machine-learning.list'

sudo apt-get update

# Install development and runtime libraries (~4GB)
sudo apt-get install --no-install-recommends \
    cuda-10-1 \
    libcudnn7=7.6.4.38-1+cuda10.1  \
    libcudnn7-dev=7.6.4.38-1+cuda10.1


# Install TensorRT. Requires that libcudnn7 is installed above.
sudo apt-get install -y --no-install-recommends libnvinfer6=6.0.1-1+cuda10.1 \
    libnvinfer-dev=6.0.1-1+cuda10.1 \
    libnvinfer-plugin6=6.0.1-1+cuda10.1

I think this thread summarizes very well what it’s like working with cuda/tensorflow

I have the same problem. How to slove it? Too many answers confuse me .Can someone summarize the solution to this problem?

That is correct. if you install tensorrt, the problem should go away. But otherwise, TF should continue running.

@sharonwoo

Try ls /usr/local/cuda-10.0/lib64 or ls /usr/local/cuda/lib64 and see if the libraries are found

and try exporting the path export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64

Now try tf.test.is_gpu_available()

The above worked for me and my tensorflow-gpu version is 2.0

@mjlbach yes, reporting for tf 2.1.

I downgraded to 2.0 to get a working version of tf, and added in the note as info to anyone coming here after encountering the error. I have edited the wording to make it less confusing.

Had the same issue on Linux, Ubuntu 18.04.3, using Python 3.7.3, within virtual environment, when running in the console :

python -c; import keras;print(keras.__version__)

The problem occurs when trying to import keras or tensorflow.

I had to add --allow-downgrade flag to apt-get to install Tensorflow RT libnvinfer6. I.e.:

sudo apt-get install --no-install-recommends \
    cuda-10-1 \
    libcudnn7=7.6.4.38-1+cuda10.1  \
    libcudnn7-dev=7.6.4.38-1+cuda10.1

sudo apt-get install -y --allow-downgrades --no-install-recommends \
        libnvinfer6=6.0.1-1+cuda10.1 \
        libnvinfer-dev=6.0.1-1+cuda10.1 \
        libnvinfer-plugin6=6.0.1-1+cuda10.1

The problem went away.

2 hours into troubleshooting this error. What the hell is TensorRT and why do we even need it? Seems like more trouble than it’s worth to me.

I get the same problem as well. Had to downgrade to 2.0.

I am using anaconda with Ububtu and tensorflow-2.1, as mentioned above

• pip install tensorflow-gpu==2.0.0
• pip install tensorflow==2.0.0

worked for me too

@Black-Behemoth Tensorflow 2.1 is no longer split into a gpu and non-gpu package

Does not explain why “import tensorflow” works in tensrflow 2.0 but attempts to invoke a cuda library in tensorflow 2.1. On a laptop with an AMD card no less.

it is a warning, i found that TensorRT is a nVidia developed library to improve the CNN performance, but i think it is not needed, I’m working with tensorflow 2.0 in Colab and it is working very good.

You can find a tutorial here: https://medium.com/@ardianumam/installing-tensorrt-in-ubuntu-dekstop-1c7307e1dcf6

I get the same problem, but because other reason:

2020-01-21 16:37:57.869790: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /usr/lib/x86_64-linux-gnu/libnvinfer.so.6)
2020-01-21 16:37:57.870308: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /usr/lib/x86_64-linux-gnu/libnvinfer.so.6)

It is because GLIBC.

Just to reiterate, this is a problem with tensorflow 2.1, not nightly or 2.0.

@gnthibault tensorflow and tensorflow-gpu now point to the same package as of 2.1

@behdadforghani-cbre’s solution to add --allow-downgrades , solved my issue in ubuntu 18.04

@jchwenger I installed all my environment using pip install, and when i tried to execute my program i had the same issue, so i added cudatoolkit=10.1 and cudnn=7.6.4 in my env using conda install -c anaconda cudatoolkit=10.1 cudnn=7.6.4, and it work for me.

@gunan: @behdadforghani-cbre’s solution to add --allow-downgrades to the last batch of installs solved the issue on two Ubuntu 18.04 machines, it might be good to add it to the install page?

# Install TensorRT. Requires that libcudnn7 is installed above.
sudo apt-get install -y --allow-downgrades --no-install-recommends \
        libnvinfer6=6.0.1-1+cuda10.1 \
        libnvinfer-dev=6.0.1-1+cuda10.1 \
        libnvinfer-plugin6=6.0.1-1+cuda10.1

@Black-Behemoth Tensorflow 2.1 is no longer split into a gpu and non-gpu package

See: https://github.com/tensorflow/docs/pull/1403 . Also, still unclear why nightly is not showing this error.