tensorflow: ImportError: libnvidia-fatbinaryloader.so.375.51: cannot open shared object file

OS: Ubuntu 17.04
*TensorFlow installed from: source
TensorFlow version: 1.1.0rc1
Bazel version (if compiling from source): 0.4.5
CUDA/cuDNN version: 8.0/5.1
GPU Model and Memory: NVIDIA GTX 1060, 6 GB RAM
Exact command to reproduce:

import tensorflow as tf

Describe the problem clearly

I installed Tensorflow from source in order to have all CPU instructions supported.

(tensorflow) stefano@stefano-PC:~$ python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/stefano/tensorflow/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/home/stefano/tensorflow/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libnvidia-fatbinaryloader.so.375.51: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/__init__.py", line 51, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/stefano/tensorflow/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/home/stefano/tensorflow/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libnvidia-fatbinaryloader.so.375.51: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
>>>

My bashrc:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/nvidia-375
export LIBRARY_PATH=${LIBRARY_PATH}:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/nvidia-375

Problem is, that Tensorflow tries to load libnvidia-fatbinaryloader.so.375.51, but it exists only libnvidia-fatbinaryloader.so.375.38. I’m puzzled why TF tries to load a newer version, because right now, version 375.38 is the latest driver available!

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 58 (14 by maintainers)

Most upvoted comments

I added /usr/lib/nvidia-375 to my LD_LIBRARY_PATH and got rid of this import error @danindiana

+50

madhupkumar on May 12, 2017

Best solution is to prepend new paths to old LD_LIBRARY_PATH in ~/.bashrc as following:

export LD_LIBRARY_PATH=$CUDA_HOME/lib64:/usr/lib/nvidia-375:$LD_LIBRARY_PATH

stefanofiorentino on Jul 24, 2017

@sudoankit Environment variables solved the problem for me on ubuntu 16.04

nvidia-375.66 TF 1.1.0

kapilgarg1996 on May 26, 2017

Had the same import issue. Before you do the import I suggest to check if all these apply:

OS: Ubuntu 16.04 LTS
Driver: nvidia-375, Secure Boot should be off.
cuDNN: cuDNN v5.1 (Jan 20, 2017), for CUDA 8.0
I recommend building TensorFlow from source using bazel and then pip wheel.
I also don’t recommend Anaconda for TensorFlow.
.bashrc or zshconfig :

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/lib/nvidia-375:/usr/local/cuda/extras/CUPTI/lib64"
export CUDA_HOME=/usr/local/cuda

All the best.

sudoankit on Aug 25, 2017

I fixed the error by using a symbolic link.

ln -s libnvidia-fatbinaryloader.so.375.66 libnvidia-fatbinaryloader.so.375.51

royshan on Jun 3, 2017

Just because I’m at the same place at the same time as you guys, I’ll comment something non-obvious I found. Many hours were sacrificed for this knowledge.

You might think cuda can either be installed with a .deb file or a .run file. HOWEVER, if you want to use the slightly older 375.39, you need to use the run file. This annoyed me because I think the rule of thumb is not to mix run files and deb files with drivers.

The reason is that the run file will prompt you whether you want to upgrade the driver during installation. Whereas during the procedure: dpkg -i <cuda.deb>&& apt-get update && apt-get install cuda, there is no choice in the matter. Upgrading to 375.51 is forced. You can try to insist not to upgrade the driver with apt-get --nodeps or dpkg --ignore-dep=nvidia-375, but then apt-get will complain of broken dependencies.

I am currently exploring alternatives. My system is ubuntu16.04, 2*TitanX (pascal).

22csnyder on Apr 19, 2017

In my new eGPU setup with Ubuntu 17.10 with TF 1.6 (pip) with a GTX1070Ti and nVidia driver 387.34, I ran into the same error just some minutes ago. Even that it was working fine yesterday. I could simply solve it with this command:

$ prime-select nvidia

As far as I know, this is something like manual nVidia Optimus, where I have to select the external GPU. I have to run this command every time before I can use the eGPU. Maybe this is just an eGPU specific solution, but at least it worked in my case and I wanted to share this approach.

b3nk4n on Mar 29, 2018

What workaround? I don’t see one. I had exported all relevant path variables. See my first post. And btw: A workaround means, that users still will encounter this issue and will maybe find by luck this issue.

StefanoD on May 22, 2017

@madhupkumar

17:27:32 CDT UPDATE: OK. After installing cuDNN and exporting the LD_LIBRARY_PATH I was able to get it working. Thanks for your help.

Oops. OK. It appears I didn’t install libcudnn5 so let me see if that works and eliminates the error. Google’s Tensorflow install instructions don’t mention that unfortunately.

Can you be a little more specific? I seem to have added that to my LD_LIBRARY_PATH as well but am still getting an error. Thanks for the help.

echo $LD_LIBRARY_PATH /home/speed/torch/install/lib:/home/speed/torch/install/lib:/home/speed/torch/install/lib:/home/speed/torch/install/lib::/usr/lib/nvidia-375

This is what my bashrc has in it:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/nvidia-375 export LIBRARY_PATH=${LIBRARY_PATH}:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/nvidia-375

Perhaps there is a conflict with minconda3 linking?

added by Miniconda3 3.10.1 installer

export PATH=“/home/speed/miniconda3/bin:$PATH” source /home/speed/perl5/perlbrew/etc/bashrc export PATH=“/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games: /usr/local/games” export PATH=“$PATH:$HOME/bin”

export NVM_DIR=“/home/speed/.nvm” [ -s “$NVM_DIR/nvm.sh” ] && . “$NVM_DIR/nvm.sh” # This loads nvm

. /home/speed/torch/install/bin/torch-activate

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib/nvidia-375 export LIBRARY_PATH=${LIBRARY_PATH}:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/nvidia-375

danindiana on May 12, 2017

@EMCP last thing to check is going with:

find / -name 'libnvidia-*' 2>/dev/null | xargs ldd | grep "not found"

that lets you know if reinstalling/upgrading the driver version left something broken (decina’s 10-tin-underpants principle)

stefanofiorentino on Sep 7, 2017

Hi, I didn’t downgraded the driver, but the OS from 17.04 to 16-04 and then it worked. @Songweiping used Ubuntu 14.04 LTS and had the same issue.

It seems TF can’t handle different point releases of the driver.

I still have the same driver version on 16.04 (which strangely works):

stefano@stefano-linux:~$ ls -l /usr/lib/nvidia-375
insgesamt 120540
-rw-r--r-- 1 root root        0 Mär 17 11:30 alt_ld.so.conf
drwxr-xr-x 2 root root     4096 Apr 13 18:23 bin
-rw-r--r-- 1 root root       42 Mär 17 11:30 ld.so.conf
lrwxrwxrwx 1 root root       23 Mär 17 11:31 libEGL_nvidia.so.0 -> libEGL_nvidia.so.375.39
-rw-r--r-- 1 root root  1079216 Feb  1 06:06 libEGL_nvidia.so.375.39
lrwxrwxrwx 1 root root       11 Mär 17 11:31 libEGL.so -> libEGL.so.1
-rw-r--r-- 1 root root    77264 Feb  1 04:35 libEGL.so.1
-rw-r--r-- 1 root root    20608 Feb  1 04:35 libEGL.so.375.39
-rw-r--r-- 1 root root   822584 Feb  1 04:35 libGLdispatch.so.0
lrwxrwxrwx 1 root root       29 Mär 17 11:31 libGLESv1_CM_nvidia.so.1 -> libGLESv1_CM_nvidia.so.375.39
-rw-r--r-- 1 root root    51528 Feb  1 06:06 libGLESv1_CM_nvidia.so.375.39
lrwxrwxrwx 1 root root       17 Mär 17 11:31 libGLESv1_CM.so -> libGLESv1_CM.so.1
-rw-r--r-- 1 root root    39048 Feb  1 04:35 libGLESv1_CM.so.1
lrwxrwxrwx 1 root root       26 Mär 17 11:31 libGLESv2_nvidia.so.2 -> libGLESv2_nvidia.so.375.39
-rw-r--r-- 1 root root    81744 Feb  1 06:06 libGLESv2_nvidia.so.375.39
lrwxrwxrwx 1 root root       14 Mär 17 11:31 libGLESv2.so -> libGLESv2.so.2
-rw-r--r-- 1 root root    51848 Feb  1 04:35 libGLESv2.so.2
lrwxrwxrwx 1 root root       10 Mär 17 11:31 libGL.so -> libGL.so.1
lrwxrwxrwx 1 root root       14 Mär 17 11:31 libGL.so.1 -> libGL.so.1.0.0
-rw-r--r-- 1 root root   583912 Feb  1 04:35 libGL.so.1.0.0
lrwxrwxrwx 1 root root       23 Mär 17 11:31 libGLX_indirect.so.0 -> libGLX_nvidia.so.375.39
lrwxrwxrwx 1 root root       23 Mär 17 11:31 libGLX_nvidia.so.0 -> libGLX_nvidia.so.375.39
-rw-r--r-- 1 root root  1278328 Feb  1 04:36 libGLX_nvidia.so.375.39
lrwxrwxrwx 1 root root       11 Mär 17 11:31 libGLX.so -> libGLX.so.0
-rw-r--r-- 1 root root    64688 Feb  1 04:35 libGLX.so.0
lrwxrwxrwx 1 root root       15 Mär 17 11:31 libnvcuvid.so -> libnvcuvid.so.1
lrwxrwxrwx 1 root root       20 Mär 17 11:31 libnvcuvid.so.1 -> libnvcuvid.so.375.39
-rw-r--r-- 1 root root  2277072 Feb  1 05:50 libnvcuvid.so.375.39
lrwxrwxrwx 1 root root       18 Mär 17 11:31 libnvidia-cfg.so -> libnvidia-cfg.so.1
lrwxrwxrwx 1 root root       23 Mär 17 11:31 libnvidia-cfg.so.1 -> libnvidia-cfg.so.375.39
-rw-r--r-- 1 root root   157896 Feb  1 05:47 libnvidia-cfg.so.375.39
lrwxrwxrwx 1 root root       23 Mär 17 11:31 libnvidia-compiler.so -> libnvidia-compiler.so.1
lrwxrwxrwx 1 root root       28 Mär 17 11:31 libnvidia-compiler.so.1 -> libnvidia-compiler.so.375.39
-rw-r--r-- 1 root root 48276880 Feb  1 06:08 libnvidia-compiler.so.375.39
-rw-r--r-- 1 root root 27269200 Feb  1 06:06 libnvidia-eglcore.so.375.39
-rw-r--r-- 1 root root    29872 Feb  1 06:06 libnvidia-egl-wayland.so.375.39
lrwxrwxrwx 1 root root       21 Mär 17 11:30 libnvidia-encode.so -> libnvidia-encode.so.1
lrwxrwxrwx 1 root root       26 Mär 17 11:30 libnvidia-encode.so.1 -> libnvidia-encode.so.375.39
-rw-r--r-- 1 root root   170192 Feb  1 05:50 libnvidia-encode.so.375.39                                                                                                                                 
-rw-r--r-- 1 root root   308296 Feb  1 04:50 libnvidia-fatbinaryloader.so.375.39                                                                                                                        
lrwxrwxrwx 1 root root       18 Mär 17 11:30 libnvidia-fbc.so -> libnvidia-fbc.so.1                                                                                                                     
lrwxrwxrwx 1 root root       23 Mär 17 11:30 libnvidia-fbc.so.1 -> libnvidia-fbc.so.375.39                                                                                                              
-rw-r--r-- 1 root root    94440 Feb  1 04:41 libnvidia-fbc.so.375.39                                                                                                                                    
-rw-r--r-- 1 root root 29077624 Feb  1 04:35 libnvidia-glcore.so.375.39                                                                                                                                 
-rw-r--r-- 1 root root   488800 Feb  1 05:51 libnvidia-glsi.so.375.39                                                                                                                                   
lrwxrwxrwx 1 root root       18 Mär 17 11:30 libnvidia-ifr.so -> libnvidia-ifr.so.1                                                                                                                     
lrwxrwxrwx 1 root root       23 Mär 17 11:30 libnvidia-ifr.so.1 -> libnvidia-ifr.so.375.39                                                                                                              
-rw-r--r-- 1 root root   207880 Feb  1 06:06 libnvidia-ifr.so.375.39                                                                                                                                    
lrwxrwxrwx 1 root root       17 Mär 17 11:31 libnvidia-ml.so -> libnvidia-ml.so.1
lrwxrwxrwx 1 root root       22 Mär 17 11:31 libnvidia-ml.so.1 -> libnvidia-ml.so.375.39
-rw-r--r-- 1 root root  1197664 Feb  1 05:48 libnvidia-ml.so.375.39
-rw-r--r-- 1 root root  9158056 Feb  1 04:55 libnvidia-ptxjitcompiler.so.375.39
-rw-r--r-- 1 root root    13080 Feb  1 04:40 libnvidia-tls.so.375.39
lrwxrwxrwx 1 root root       23 Apr 13 18:23 libnvidia-wfb.so.1 -> libnvidia-wfb.so.375.39
-rw-r--r-- 1 root root   295416 Dez  7  2012 libnvidia-wfb.so.375.39
lrwxrwxrwx 1 root root       14 Mär 17 11:31 libOpenGL.so -> libOpenGL.so.0
-rw-r--r-- 1 root root   184264 Feb  1 04:35 libOpenGL.so.0
drwxr-xr-x 2 root root     4096 Apr 13 18:23 tls
drwxr-xr-x 2 root root     4096 Apr 13 18:23 vdpau
drwxr-xr-x 2 root root     4096 Apr 13 18:23 xorg

StefanoD on May 23, 2017

I’m glad to hear it’s working. Our support matrix for Ubuntu Linux only includes versions 14 LTE to 16 LTE. Thank you for bringing the compatibility issues with 17 to the attention of the community.

jart on Apr 14, 2017