tensorflow: tensorflow-gpu: Could not load dynamic library 'libcudart.so.10.1'
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ArchLinux 64bits
- TensorFlow installed from (source or binary): binary
- TensorFlow version (use command below): tensorflow-gpu 2.2.0rc4
- Python version: 3.8.2
- CUDA/cuDNN version: 10.2.89-5 (from Archlinux repo)
Describe the problem
TensorFlow doesn’t use my GPU because there is a bug while trying to load “libcudart.so.10.1”. My system has libcudart.so.10.2 installed All the other libraries load fine since they look for libcu***.so.10 and not 10.1
Source code / logs
Python 3.8.2 (default, Apr 8 2020, 14:31:25)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
2020-05-03 22:17:42.422067: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-03 22:17:42.471297: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-03 22:17:42.472040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.683GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2020-05-03 22:17:42.472169: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-05-03 22:17:42.473728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-03 22:17:42.475267: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-03 22:17:42.475499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-03 22:17:42.477078: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-03 22:17:42.478053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-03 22:17:42.481446: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-03 22:17:42.481478: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Num GPUs Available: 0
Here is the CUDA lib I have installed:
/opt/cuda/doc/man/man7/libcudart.7
/opt/cuda/doc/man/man7/libcudart.so.7
/opt/cuda/targets/x86_64-linux/lib/libcudart.so
/opt/cuda/targets/x86_64-linux/lib/libcudart.so.10
/opt/cuda/targets/x86_64-linux/lib/libcudart.so.10.2
/opt/cuda/targets/x86_64-linux/lib/libcudart.so.10.2.89
/opt/cuda/targets/x86_64-linux/lib/libcudart_static.a
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 8
- Comments: 25 (1 by maintainers)
meet this problem when using tensorflow2.2 with cuda10.2. Solved by ln -s
sudo ln -s /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2 /usr/lib/x86_64-linux-gnu/libcudart.so.10.1
If you are using conda to manage your envs, just specify the cuda env when installing tensorflow. This way you can keep cuda 11 installed, and still run tf with an old cuda version.
conda install tensorflow-gpu cudatoolkit=10.1I am facing the same problem. Even Cuda 11 successfully working but tensorflow 2.3 looking cuda 10.1
I’m having this message with cuda 11 installed. As I know cuda 11 should be backward compatible with everything using cuda 10. Does it have any negative effects? It also shows “Ignore above cudart dlerror if you do not have a GPU set up on your machine.”
It seems that libcudart 10.1 and 10.2 are compatible. I was able to hack this on Ubuntu 20.04 by providing a symlink as mentioned by others and run tensorflow 2.2 with libcudart 10.2 (without building from source, just the symlink from a fake library). This may help: https://github.com/tensorflow/tensorflow/issues/38194#issuecomment-629801937
Same Here. Pytorch is working fine with GPU’s. But getting errors with tensorflow.
ln -s /usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudart.so.11.0.221 /usr/lib/x86_64-linux-gnu/libcudart.so.10.1
“sudo ln -s /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2 /usr/lib/x86_64-linux-gnu/libcudart.so.10.1” It works. Thank you stdcoutzyx.
If you have installed cuda-10.2 and still have problems, try this: cd /usr/local/cuda_10.2/lib64 ln -s libcudart.so.10.2 libcudart.so.10.1
i have same issue, @ioannist try this conda install tensorflow-gpu cudatoolkit=10.1 but still same issue
Thx, this works for me. CUDA 11.0 + python 3.8
Okay I tried and Archlinux does indeed fix the issue with the package tensorflow-cuda. I’ll keep using that and not tensorflow-gpu with pip. Thanks!