tensorflow: "Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found"

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version: 2.4.0
  • Python version: 3.8.8
  • Installed using virtualenv? pip? conda?: pip inside a virtualenv.
  • Bazel version (if compiling from source): N/A
  • GCC/Compiler version (if compiling from source): N/A
  • CUDA/cuDNN version: v11.0.3_451.82 / cudnn-11.0-windows-x64-v8.0.4.30.zip
  • GPU model and memory: 2080 8G -Nvidia driver version: 465.89

Describe the problem

A similar issue as to #46606, #43193, and #45055. When attempting to run python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))", I get the following trace;

2021-05-02 16:52:00.618495: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2021-05-02 16:52:00.618653: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-05-02 16:52:02.101881: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-05-02 16:52:02.102613: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll 2021-05-02 16:52:02.130031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:2d:00.0 name: NVIDIA GeForce RTX 2080 computeCapability: 7.5 coreClock: 1.845GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.23GiB/s 2021-05-02 16:52:02.130365: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2021-05-02 16:52:02.130609: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found 2021-05-02 16:52:02.130862: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found 2021-05-02 16:52:02.131045: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found 2021-05-02 16:52:02.131225: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found 2021-05-02 16:52:02.131484: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found 2021-05-02 16:52:02.131796: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found 2021-05-02 16:52:02.132066: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found 2021-05-02 16:52:02.132188: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... 2021-05-02 16:52:02.132804: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-05-02 16:52:02.133414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-05-02 16:52:02.133546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 2021-05-02 16:52:02.133646: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set tf.Tensor(-115.45033, shape=(), dtype=float32)

Provide the exact sequence of commands / steps that you executed before running into the problem

I continue to get the above issue despite following the steps and suggestions in each of the other similar issues.

I tried adding the environmental variables through cmd, which wasn’t mirrored in the GUI so I added them there as well. This did not fix the issue.

I re-installed the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017, and 2019. This did not fix the issue.

I enabled long paths. This did not fix the issue.

I was already on a 64bit release of Python 3.8.8, which the venv I’m using was created with.

I’ve restarted the terminal and restarted my PC several times, which did not fix the issue.

I created the C:\tools\cuda folder and extracted the contents of the cuDNN there, and ensured that it was on PATH. This confuses me since it’s not mentioned as a necessity in the installation process of cuDNN itself. Either way, this did not fix the issue.

Any other info / logs deviceQuery running. image All the items in PATH. image The contents of the CUDA bin folder, showing the supposedly missing files exist. image

Thank you in advance for your help.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 7
  • Comments: 20 (1 by maintainers)

Most upvoted comments

I was able to get past this error by doing os.add_dll_directory("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2/bin") before importing tensorflow

I was able to get past this error by doing os.add_dll_directory("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2/bin") before importing tensorflow

Thanks, it saved my day!

bumping this here because the fix looks like temporary. I did not face this issue using python 3.8 in a virtualenv but occured in system python 3.9 on windows. From the people it also looks like a windows issue. Even though the cuda paths added to path correctly, it still doesn’t recognize the appropriate dll’s.

EDIT

I can confirm the fault is on windows. If you install the python from store, python doesn’t load the path. Install it from python official website.

@karthik-653 Before impoting anything(besides os ofc) should work.

This solution (the answer on the issue) is for PYTHON. THIS IS PYTHON ISSUE. SPECIFICALLY STORE INSTALLED WHATEVER YOUR PYTHON CODE YOU ARE RUNNİNG. ADD THIS TO THE TOP OF THE FILE. E.G. IF YOU ARE RUNNING THE CODE WITH python main.py, add below code to top of the main.py file

import os
os.add_dll_directory("C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2/bin")

Change path accordingly with your cuda version.