tensorflow: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice

Click to expand!

Issue Type

Bug

Source

binary

Tensorflow Version

tf-nightly

Custom Code

Yes

OS Platform and Distribution

WSL Ubuntu

Mobile device

No response

Python version

3.9

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

11.2

GPU model and memory

No response

Current Behaviour?

Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice

The error appears when setting groups > 1 in tf.keras.layers.Conv2D with tf-nightly.

The error does not appear in TensorFlow 2.8.

Seems the previous TensorFlow build is packed with libdevice.

Standalone code to reproduce the issue

tf.keras.layers.Conv2D(
            512, 
            1,
            use_bias=False
            groups=5,
            )

Relevant log output

2022-07-28 01:48:13.943787: W tensorflow/compiler/xla/service/gpu/nvptx_helper.cc:56] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
  ./cuda_sdk_lib
  /usr/local/cuda-11.2
  /usr/local/cuda
  .
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions.  For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2022-07-28 01:48:13.986332: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2022-07-28 01:48:13.986408: W tensorflow/stream_executor/gpu/asm_compiler.cc:80] Couldn't get ptxas version string: INTERNAL: Couldn't invoke ptxas --version
2022-07-28 01:48:14.018602: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2022-07-28 01:48:14.018778: F tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:453] ptxas returned an error during compilation of ptx to sass: 'INTERNAL: Failed to launch ptxas'  If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.
Fatal Python error: Aborted

Thread 0x00007f3abe7fc640 (most recent call first):
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 316 in wait
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 574 in wait
  File "/home/edwardyehuang/.vscode-server/extensions/ms-python.python-2022.10.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 257 in _on_run
  File "/home/edwardyehuang/.vscode-server/extensions/ms-python.python-2022.10.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 930 in _bootstrap

Thread 0x00007f3abeffd640 (most recent call first):
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 316 in wait
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 574 in wait
  File "/home/edwardyehuang/.vscode-server/extensions/ms-python.python-2022.10.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 211 in _on_run
  File "/home/edwardyehuang/.vscode-server/extensions/ms-python.python-2022.10.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/home/edwardyehuang/miniconda3/envs/tf-nightly/lib/python3.9/threading.py", line 930 in _bootstrap

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 30 (13 by maintainers)

Most upvoted comments

Solved, but imperfectly.

@mohantym

The workaround is simple in my situation. (Pip installed TensorFlow and used LD_LIBRARY_PATH to help TensorFlow to find conda installed cuda and cudnn libs), which has three steps:

I install the cuda-nvcc package from conda.
Create dirs with the structure “xxxx/nvvm/libdevice”, and copy “$CONDA_PREFIX/lib/libdevice.10.bc” to “xxxx/nvvm/libdevice”
Set “export XLA_FLAGS=–xla_gpu_cuda_data_dir=xxxx”

However, this issue can be avoid in TensorFlow C++ code (tensorflow/compiler/xla/service/gpu/nvptx_helper.cc). The searching of “libdevice.10.bc” should not rely on function “GetLibdeviceDir” and the hard-coded dir structure:

std::string libdevice_dir =
        tensorflow::io::JoinPath(cuda_root, "nvvm", "libdevice");

TensorFlow should be also expected “libdevice.10.bc” is appeared in the search path, just like other cuda and cudnn files.

edwardyehuang on Sep 7, 2022

On Windows 10, with Anaconda and CUDA installed by “conda install -c conda-forge cudatoolkit=11.2”

In the root of your python project create the directory “nvvm” and “nvvm/libdevice”
From “C:\Users<user_name>\anaconda3\pkgs\cudatoolkit-11.2.2-h933977f_10\DLLs” copy the files libdevice.10.bc and nvvm64_40_0.dll" to the subdirectory “libdevice” created following the point one. It works

EarthShaping on Sep 4, 2022

Solved, but imperfectly.

@mohantym

The workaround is simple in my situation. (Pip installed TensorFlow and used LD_LIBRARY_PATH to help TensorFlow to find conda installed cuda and cudnn libs), which has three steps:

I install the cuda-nvcc package from conda.

Create dirs with the structure “xxxx/nvvm/libdevice”, and copy “$CONDA_PREFIX/lib/libdevice.10.bc” to “xxxx/nvvm/libdevice”

Set “export XLA_FLAGS=–xla_gpu_cuda_data_dir=xxxx”

However, this issue can be avoid in TensorFlow C++ code (tensorflow/compiler/xla/service/gpu/nvptx_helper.cc). The searching of “libdevice.10.bc” should not rely on function “GetLibdeviceDir” and the hard-coded dir structure:
std::string libdevice_dir =
        tensorflow::io::JoinPath(cuda_root, "nvvm", "libdevice");
TensorFlow should be also expected “libdevice.10.bc” is appeared in the search path, just like other cuda and cudnn files.

That didn’t work for me, except if xxxx is /miniconda/lib so putting nvvm/libdevice/libdevice.10.bc into /miniconda/lib everything else is the same.

altair199797 on Oct 4, 2022

This is still an issue in TF 2.12, see https://github.com/keras-team/tf-keras/issues/62 for a reproduction example.

drasmuss on Sep 22, 2023

I was having this problem to use a package with GPU, and the preceding solutions did not work. I could solve it by just downgrading tensorflow-gpu to 2.10.1.

rogeriog on Nov 25, 2022