tensorflow: Could not load dynamic library 'libcusolver.so.10' - TF-2.4.0RC, Cuda,CudNN, RTX 3080

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): pip
TensorFlow version: 2.4.0-rc1
Python version: 3.8
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 11.1 & 8.0.5
GPU model and memory: RTX 3080

Describe the problem Downloading everything per instructions, all GPU libs are being read except the one in the title. No idea why.

2020-11-11 14:48:06.269458: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2020-11-11 14:48:06.269897: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2020-11-11 14:48:06.303715: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-11-11 14:48:06.304046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1724] Found device 0 with properties: pciBusID: 0000:08:00.0 name: GeForce RTX 3080 computeCapability: 8.6 coreClock: 1.71GHz coreCount: 68 deviceMemorySize: 9.78GiB deviceMemoryBandwidth: 707.88GiB/s 2020-11-11 14:48:06.304061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2020-11-11 14:48:06.305025: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2020-11-11 14:48:06.305052: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2020-11-11 14:48:06.305398: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2020-11-11 14:48:06.305501: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2020-11-11 14:48:06.305583: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64:/usr/lib/cuda/include:/usr/lib/cuda/lib64: 2020-11-11 14:48:06.305805: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2020-11-11 14:48:06.305877: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2020-11-11 14:48:06.305883: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1761] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... 2020-11-11 14:48:06.306061: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-11-11 14:48:06.306326: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2020-11-11 14:48:06.306336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1265] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-11-11 14:48:06.306340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1271]

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 9
Comments: 33 (3 by maintainers)

Commits related to this issue

Fix could not load dynamic library 'libcusolver.so.10' error Reference: https://github.com/tensorflow/tensorflow/issues/44777#issuecomment-771285431 Command and error message: (softlearning) ro... — committed to j3soon/rail-berkeley-softlearning by j3soon 7 months ago

Most upvoted comments

cd $LD_LIBRARY_PATH
sudo ln libcusolver.so.11 libcusolver.so.10  # hard link

I was able to solve the issue with missing hard linking libcusolver.so.11 with libcusolver.so.10. (CUDA 11.2, RTX3070s)

+63

JihoChoi on Feb 3, 2021

One workaround is to install libcusolver-11-0 side by side with cuda-11-1 or cuda-11-2 and tweak your LD_LIBRARY_PATH. For example, to get TensorFlow 2.4 to work with CUDA 11.2:

$ apt-get install cuda-11-2 libcusolver-11-0
$ export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.0/lib64

+18

reuben on Jan 27, 2021

@cameronjacobson good catch! The hard link didn’t work for me but your approach gave me the hint to add the soft link in my virtual environment.

cd /.virtualenvs/myvenvname/lib/python3.8/site-packages/tensorflow/python
ln -s /path/to/my/cuda/lib/libcusolver.so.11 libcusolver.so.10

System: cuda 11.2, cuDNN 8.2, GTX1080, pop_OS 20.04 (ubuntu 20.04)

diplodocuslongus on Apr 28, 2021

I have the same problem with tf-nightly, CUDA 11.1.1 and cudnn 8.0.5 on Ubuntu. Edit: #43947 (comment) That solution worked for me.

Can you post your set up also? with dependencies for CUDA version and CUdNN version ?

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): pip
TensorFlow version: tf_nightly
Python version: 3.8.5
Installed using virtualenv? pip? conda?: conda
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 11.1.1 & 8.0.5
GPU model and memory: RTX 3080

I think sudo ln creates a link from libcusolver.so.11 (which is not yet supported) to libcusolver.so.10, so that tensorflow finds it. But I don’t have much knowledge on that part. Just guessing and trying. For me it worked and my RTX3080 now runs Tensorflow.

nilskk on Nov 13, 2020

cd $LD_LIBRARY_PATH
sudo ln libcusolver.so.11 libcusolver.so.10  # hard link
I was able to solve the issue with missing hard linking libcusolver.so.11 with libcusolver.so.10. (CUDA 11.2, GetForce 2060)

This works perfectly,just make sure you have set LD_library path exactly where libcusolver.so.11 is present and make the hard link… check your LD_library_path

Thanks

RamKaistha on Apr 8, 2021

I have cuda 11.1 and cudnn 8.0.5 installed, while as suggested by TensorFlow Tested build configurations, we should have cuda 11.0 when installing TF 2.4. This should be that reason. If you are using Conda, a workaround is to:

conda install cudatoolkit=11.0.221 cudnn=8.0.4 -c nvidia

forskamse on Jan 18, 2022

apt install nvidia-cuda-toolkit solved the issue for me

marekasf on May 12, 2021

cd $LD_LIBRARY_PATH
sudo ln libcusolver.so.11 libcusolver.so.10  # hard link
I was able to solve the issue with missing hard linking libcusolver.so.11 with libcusolver.so.10. (CUDA 11.2, RTX3070s)

For me, this does not work. Making a link to the tensorflow library instead of $LD_LIBRARY_PATH solves my problem.

ln -s /usr/local/cuda/lib64/libcusolver.so.11 (venv dir)/lib/python3.6/site-packages/tensorflow/python/libcusolver.so.10

In my case

ln -s /usr/local/cuda/lib64/libcusolver.so.11 ~/.local/share/virtualenvs/my-project/lib/python3.7/site-packages/tensorflow/python/libcusolver.so.10

Either soft or hard link works.

(CUDA 11.2, GTX970, debian 10)

HYChou0515 on Apr 11, 2021

The only workaround which is consistent with 2.4 seems to be to uninstall 11.2 and install 11.0. Nothing else resolves the “Could not load dynamic library ‘libcusolver.so.10’; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory”

This is not a tensorflow bug – it is an NVIDIA bug.

JMerrill-Fairness on Jan 23, 2021

This appears to still be an issue, but maybe everyone is finding different ways around the problem.

Running an strace showed me that tensorflow is trying to find “libcusolver.so.10” under “~/.local”, and not in its (probably expected) location, since none of the other shared libs are here. Could have something to do with the fact that I installed tensorflow via a “local” install not system-level install.

After I put soft link here everything worked fine: ~/.local/lib/python3.7/site-packages/tensorflow/python/libcusolver.so.10

While troubleshooting I found a different, but maybe already reported issue. keras will complain if I try to run tf-nightly. It doesn’t seem to recognize that the tf-nightly version is above its required version.

cameronjacobson on Mar 22, 2021

I am also having this issue with a brand new installation of TF2.4 - was this left hanging somehow?

I think a solution might be to install cuda 10 and add libcusolver to your path…

Symlinking libcusolver.so.11 to .so.10 does NOT work, though this issue gives it as a solution:

https://github.com/tensorflow/tensorflow/issues/43947

What finally worked for me was installing CUDA11.0 - seems any higher versions are not supported.

ckyleda on Jan 14, 2021