i install cuda 12.3.2 and cudnn .then when i install tensorflow via pip install tensorflow[and-cuda] will auto installtensorflow 2.16.1 and can not find my GPU. when i input pip install tensorflow==2.15 . It will autoremove 2.16.1 and install 2.15.then I can find my GPU

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

bash output

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-03-13 02:41:48.797545: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-13 02:41:48.797600: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-13 02:41:48.798179: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-13 02:41:48.801510: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-13 02:41:49.293727: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-13 02:41:49.828549: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-13 02:41:49.934333: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-13 02:41:49.934403: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:887] could not open file to read NUMA node: /sys/bus/pci/devices/0000:08:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Almost final and automated fix below

• Where I found the resolution

  • TF 2.16.1 Fails to work with GPUs
    • Solution proposed by “sh-shahrokhi”, improved by “ChristofKaufmann”
    • See specially Comment by COntributor
  • Related Issues
    • GPU not detected on WSL2, where I have post some comments
    • Tensorflow WSL GPU CUDA recognition issue RTX3090
    • Once gain: tf.2.16.1 fails to recognize GPUs
  • Other mention on social media
    • Reddit issue

• Exact solution

  • Temporary fix (after activating environment in which Tensorflow 2.16.1 is installed)

      export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
      export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
      

  • Automating the variable set/unset process with Anaconda (one-time setup)
    • Activate your environement in which TF 2.16.1 is installed
    • Two files to be created in “anaconda3/envs/<ENV_NAME>/etc/conda”
      • anaconda3/envs/<ENV_NAME>/etc/conda/activate.d/env_vars.sh

        #!/bin/sh
        export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
        export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
        

      • anaconda3/envs/<ENV_NAME>/etc/conda/deactivate.d/env_vars.sh

        #!/bin/sh
        unset NVIDIA_DIR
        unset LD_LIBRARY_PATH
        

    • Official documentation to do this via conda.io
    • Stack-overflow question where I got this Set environment vars when activating conda env

• What else helped me

  • How to list all variables names and their current values?
  • Conda documenation for “env_vars.sh” for activate/deactivate

@Zhaopudark thank you for sharing. what path shoud I set for those environment variables ? Make sure $Env:CUDA_PATH is given correctly Make sure $Env:LD_LIBRARY_PATH is given correctly Make sure $Env:TF_CUDA_PATHS is given correctly

That depends on your specific installation method of cuda and cudnn. For me, since I use miniconda and conda Installation, so my setting is as:

# pwsh
# if `conda activate my_env `
# an env variable `$Env:CONDA_PREFIX` will be `/home/your_name/miniconda3/envs/my_env `
# So, the setting is as:
$Env:CUDA_PATH = $Env:CONDA_PREFIX
$Env:LD_LIBRARY_PATH="$Env:CONDA_PREFIX/lib"
$Env:TF_CUDA_PATHS="$Env:CONDA_PREFIX"

Actually, either package manager installation or conda Installation, the $Env:CUDA_PATH should be such a root path where you can see cuda’s installed directories, such as ./nvvm, ./nvvm, ./extras/Debugger/lib64/libcudacore.a and so on. And, $Env:TF_CUDA_PATHS should be equal to $Env:CUDA_PATH.

Does your setup detect GPU with tensorflow 2.16.1? mine works with 2.15 but not this one.

Thanks @zjm008 and @Zhaopudark for providing with workarounds. I certainly agree that these are ways to mitigate this problem

However, while following the pip-installation guide I can see that

pip install tensorflow[and-cuda]

do download many cuda libraries including cuDNN as well, which was set to path variables as soon as you activate a conda environment which is a very nice functionality up to TF 2.15

I believe the developers might’ve missed the same while moving to TF 2.16

Do let me know if you two agree?

@chaudharyachint08. This weekend, I have shared my steps in this post. Maybe you can check it and transfer it to your own environment.

@Zhaopudark Thank you for your comment.

Regarding the cudnn and cuda version, I searched around and finally found a matrix here.

I will update my result once I have installed newer cuda and cudnn.

@Zhaopudark thank you for sharing. what path shoud I set for those environment variables ? Make sure $Env:CUDA_PATH is given correctly Make sure $Env:LD_LIBRARY_PATH is given correctly Make sure $Env:TF_CUDA_PATHS is given correctly

That depends on your specific installation method of cuda and cudnn. For me, since I use miniconda and conda Installation, so my setting is as:

# pwsh
# if `conda activate my_env `
# an env variable `$Env:CONDA_PREFIX` will be `/home/your_name/miniconda3/envs/my_env `
# So, the setting is as:
$Env:CUDA_PATH = $Env:CONDA_PREFIX
$Env:LD_LIBRARY_PATH="$Env:CONDA_PREFIX/lib"
$Env:TF_CUDA_PATHS="$Env:CONDA_PREFIX"

Actually, either package manager installation or conda Installation, the $Env:CUDA_PATH should be such a root path where you can see cuda’s installed directories, such as ./nvvm, ./nvvm, ./extras/Debugger/lib64/libcudacore.a and so on. And, $Env:TF_CUDA_PATHS should be equal to $Env:CUDA_PATH.