tensorflow: TF 2.17.0 RC0 Fails to work with GPUs (and TF 2.16 too)
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
binary
TensorFlow version
TF 2.16.1
Custom code
No
OS platform and distribution
Linux Ubuntu 22.04.4 LTS
Mobile device
No response
Python version
3.10.12
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
12.4
GPU model and memory
No response
Current behavior?
I created a python venv in which I installed TF 2.16.1 following your instructions: pip install tensorflow When I run python, import tf, and issue tf.config.list_physical_devices(‘GPU’) I get an empty list [ ]
I created another python venv, installed TF 2.16.1, only this time with the instructions:
python3 -m pip install tensorflow[and-cuda]
When I run that version, import tensorflow as tf, and issue
tf.config.list_physical_devices(‘GPU’)
I also get an empty list.
BTW, I have no problems running on my box TF 2.15.1 with GPUs. Julia also works just fine with GPUs and so does PyTorch. the
Standalone code to reproduce the issue
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-03-09 19:15:45.018171: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-09 19:15:50.412646: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> tf.__version__
'2.16.1'
tf.config.list_physical_devices('GPU')
2024-03-09 19:16:28.923792: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-09 19:16:29.078379: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
>>>
Relevant log output
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 53
- Comments: 173 (7 by maintainers)
Links to this issue
- Tensorflow does not recognize my GPU - General Discussion - Google AI Developers Forum
- Help needed: TensorFlow Not detecting my GPU - General Discussion - Google AI Developers Forum
- Can't get Tensorflow GPU to work on WSL 2 - General Discussion - Google AI Developers Forum
- Tensorflow Installation issue, GPU not found, Ubuntu, NVidia RTX3050 - TensorFlow - Google AI Developers Forum
- Cannot install tensorflow on Anaconda/miniconda with CUDA 12.5 - General Discussion - Google AI Developers Forum
- Why is enabling CUDA support on WSL harder than actual development with tensorflow - General Discussion - Google AI Developers Forum
- TensorFlow version:2.16 just released! - Announcements - Google AI Developers Forum
- I need help. I'm not able to install Tensorflow with GPU support and I don't really know what else to do - General Discussion - Google AI Developers Forum
Commits related to this issue
- Add back `xla/stream_executor:cuda_platform` to `tf_additional_binary_deps`. Should fix https://github.com/tensorflow/tensorflow/issues/63362 Reverts changelist 582804278 PiperOrigin-RevId: 6461469... — committed to tensorflow/tensorflow by tensorflower-gardener 2 years ago
- Add back `xla/stream_executor:cuda_platform` to `tf_additional_binary_deps`. Should fix https://github.com/tensorflow/tensorflow/issues/63362 Reverts changelist 582804278 PiperOrigin-RevId: 6461469... — committed to tensorflow/tensorflow by tensorflower-gardener 2 years ago
- Add back `xla/stream_executor:cuda_platform` to `tf_additional_binary_deps`. Should fix https://github.com/tensorflow/tensorflow/issues/63362 Reverts changelist 582804278 PiperOrigin-RevId: 6461828... — committed to tensorflow/tensorflow by tensorflower-gardener 2 years ago
- Add back `xla/stream_executor:cuda_platform` to `tf_additional_binary_deps`. Should fix https://github.com/tensorflow/tensorflow/issues/63362 Reverts changelist 582804278 PiperOrigin-RevId: 6461828... — committed to tensorflow/tensorflow by tensorflower-gardener 2 years ago
- Add back `xla/stream_executor:cuda_platform` to `tf_additional_binary_deps`. Should fix https://github.com/tensorflow/tensorflow/issues/63362 Reverts changelist 582804278 PiperOrigin-RevId: 6461828... — committed to tensorflow/tensorflow by tensorflower-gardener 2 years ago
It’s just tensorflow can’t see the Cuda libraries.
Instal tensorflow[and-cuda] and add this to your .bashrc or conda activation script. Adjust python version in it according to your setup.
NVIDIA_PACKAGE_DIR=“$CONDA_PREFIX/lib/python3.12/site-packages/nvidia”
for dir in $NVIDIA_PACKAGE_DIR/*; do if [ -d “$dir/lib” ]; then export LD_LIBRARY_PATH=“$dir/lib:$LD_LIBRARY_PATH” fi done
You won’t need to install cuda or cudnn on the system. only the cuda libraries that are installed with $ pip install tensorflow[and-cuda] would be enough.
On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas @.***> wrote:
Hi @JuanVargas ,
For GPU package you need to ensure the installation of CUDA driver which can be verified with nvidia-smi command. Then you need to install TF-cuda package with
pip install tensorflow[and-cuda]which automatically installs required cuda/cudnn libraries.I have checked in colab and able to detect GPU.Please refer attached gist.
I am closing this (unresolved issue) because I am told by the Keras/TF team that the issue is related to TF.
Well, I wasted 8hr of my Sunday on this setting up another pc from scratch. Before reverting to the old version. Now looking to move off tensor flow.
Almost final and automated fix below
Where I found the resolution
Exact solution
export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))) export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}#!/bin/sh export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))) export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}What else helped me
Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation (
[environment]/etc/activate.d/env_vars.sh).This is not a resolution as this post install step should not be necessary.
W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRTI can’t seem to do similar tricks to resolve the TensorRT issues when installed similarly into the conda environment. Any ideas?
Following on from the post by chaudharyachint08, I did the following to automate it on venv.
I editied
bin/activatein the folder of my venv and added the two lines at the end of the file:Then while editing the same file, added the 2 unset lines inside the deactivate function (before the } closed curly bracket):
I had tested it by entering the 2 lines in the terminal and my GPU was detected, so this was just the automation when the venv is activated.
got it work 😃 first https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
then download Local Installer for Ubuntu22.04 x86_64 (Deb)
unpack and install libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb
I think the expected solution would be a new release that fixes this issue, so setting LD_LIBRARY_PATH is not needed like it is done in 2.15.1, it would be downgrade for users to do such workarounds it should just work with: pip install tensorflow[and-cuda]
@joaomh If you want to use 2.16.1 and you use conda then the easiest fix is with activated env:
mkdir -p ${CONDA_PREFIX}/etc/conda/activate.dcreate file in ${CONDA_PREFIX}/etc/conda/activate.d/fix_gpu.sh with content below:I have the same problem with Ubuntu 22.04.4 with the following environment:
tensorflow==2.16.1Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linuxgcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0nvcc --versionoutput:I don’t actually use TensorRT, but I would check if the required .so file for it is visible to tensorflow. Maybe I would need to find the name of required file in tensorflow source code.
This actually doesn’t change the fact that the new tensorflow version should be tested by google team before release, or the bugs should be fixed. It seems they only care about having a working docker image, not anything else.
@niko247 undoubtedly true. It is crystal clear that TF 2.16.1 does not work with the simple
pip install tensorflow[and-cuda]command to actually utilize CUDA locally and no relative guidelines where provided yet to resolve this.It seems practically impossible for someone owning a PC with CUDA-enabled GPU to perform deep learning experiments with TensorFlow version 2.16.1 and utilize his GPU locally without manually performing some extra steps not included (until today) in the official TensorFlow documentation of the standard installation procedure of TensorFlow for Linux users with GPUs at least as a temporal fix! That’s why I submitted the pull request in good faith and for the shake of all users as TensorFlow is "An Open Source Machine Learning Framework for Everyone".
Hope that the next patch version of TensorFlow will fix the bug as soon as possible!
Thanks! Downgrade to 2.15.1 works well for me too (with TF prob <=0.23).
Agree completely with the comments regarding the frustrating situation with tf 2.16.x . No wonder why torch is gaining the race of adoption
On Wed, May 15, 2024, 1:18 AM Sotiris Gkouzias @.***> wrote:
Hi Krzysztof
I visited the site https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
where I found an entry listed as " Local Installer for UBuntu22.04 x86_64(Deb)" which I downloaded. Unfortunately what I got is a package named “cudnn-local-repo-ubuntu2204-8.9.7.29_1.0-1_amd64.deb” which is not the same as the name you suggest in your message, which is " libcudnn8_8.9.7.29-1+cuda12.2_amd64.deb"
I assume what you meant is to get the libcudnn8_8.9.7.29*amd64.deb and the cuda12.2_amd64.deb separately and install both.
I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work with older versions of CUDA (12.2 or 12.3) because sooner or later the TF team will have to produce a version with the updated version of CUDA. IMHO, rather than us wasting time going back in versions, the TF beak should invest time going forward to update TF to the current CUDA version.
Thank you, Juan
On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski < @.***> wrote:
I did some tests and the model’s accuracy really dropped, but I think it’s related to keras:
Just tried it, I can confirm that 2.17.0rc1 is able to find and use the GPU.
There are new non-fatal errors logged, not sure if this is intentional:
Also, it’d be great if everyone in the community wasn’t subjected to whatever pettiness this is:
@rednag as I understand it you have two available options:
1) Keep TensorFlow version
2.15and reinstall Keras 3 afterwardsAccording to the official Keras documentation you can simply:
pip install --upgrade kerasafter installing tensorflow version2.152) Upgrade Tensorflow to version
2.16.1You can upgrade to TensorFlow version
2.16.1and utilize your GPU locally (Keras 3.0 will be installed as well) through following the steps below:pip install --upgrade pip,pip install tensorflow[and-cuda],Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIXEnter that directory and create these subdirectories and files:
Edit
./etc/conda/activate.d/env_vars.shas follows:Edit
./etc/conda/deactivate.d/env_vars.shas follows:python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"I have submitted the respective pull request to update the official TensorFlow installation guide and is currently pending review.
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!
@thephet it is a path issue. You must manually configure the environment variables.
You can try the following:
pip install --upgrade pip,pip install tensorflow[and-cuda],Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIXEnter that directory and create these subdirectories and files:
Edit
./etc/conda/activate.d/env_vars.shas follows:Edit
./etc/conda/deactivate.d/env_vars.shas follows:Verify the GPU setup:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"I have submitted the respective pull request to update the official TensorFlow installation guide a month ago.
I hope it helps!
@tknuutila and @mihaimaruseac I agree with you totally! IMHO, at least the Tensorflow team should suggest a temporal solution and change accordingly the
pip install tensorflow[and-cuda]guidelines.It is surely a pity by all means! And when I read the comment below from mr. Álvaro Rodríguez I get really disappointed because I really love TensorFlow and Keras 3.0 :
Until today the officially documented TensorFlow standard installation procedure for Linux users with GPUs does not include the additional steps required to perform deep learning experiments with TensorFlow version 2.16.1 and utilize GPU locally. That’s why I submitted the pull request in good faith and for the shake of all users as TensorFlow is “An Open Source Machine Learning Framework for Everyone”.
Hope that the next patch version of TensorFlow will fix the bug as soon as possible!
Hello everyone, Had the same problem here, Managed to solve it! Thanks to @njzjz , setting
TF_CPP_MAX_VLOG_LEVEL=3shows more information:what I did, was searching for where cudnn is placed:
adding
/usr/lib/x86_64-linux-gnutoLD_LIBRARY_PATHsolved my problem 😃Hope this helps
Not really a solution, but clarification and summary of what is happening so far. As per the investigations of @njzjz, we see that
libcublaset al are loaded, whereaslibcudnnis not. He also investigated inside a docker container and found that none of the libraries are loaded. He further updated the library path and found that the libraries were now loaded.I have tried on my end and came to the same conclusion. I think what’s happening is that I have installed
libcublas,libcudart, etcetera, from the Nvidia Fedora repo (which does not includelibcudnn), which is why these libraries load but notlibcudnnwhen I run afterpip install tensorflow[and-cuda]. Even though all these libs exist in my venv lib dir. It seems that the libraries in/usr/local/cuda/lib64are searched, but not the ones in the venv lib dir.@njzjz’s and @sgkouzias’s solutions further support this. Clearly the current workaround is to follow their path altering advice.
I hope that this is useful to somebody. Hopefully there is a fix soon.
I have given up on TensorRT. I guess I won’t be using it either.
Agreed. Installing TF has always been hit or miss and it seems that in the many years since I last used TF that hasn’t changed one bit.
Yeah, this isue can be closed as the GPU setup issues have been resolved. Will close it to signal this.
Thank you everyone for resolving this and @sgkouzias for testing every RC and final release to check for the fix!
But you still have the same ridiculous error messages so this will still cause endless headaches for those who doesn’t magically know what error messages are ok or not. It seems like no one is doing even the slightest attempt to make this work according to the install instructions! The 2.17.0rc0 at least got rid of some of the messages but they are back again in 2.17.0rc1. Version 2.17.0 should never have been released!
As I said with 2.17.0rc1 and it fails to register the cudnn, cufft, cublas… 2.17.0rc0 does not. Both have the NUMA node problem.
Now there are people who think this is acceptable because it is not a fatal error. Same with the NUMA node problem. But if those are considered acceptable for whatever reason then just mute them or at least write something in the install instructions like “any error of this nature should be ignored” so we don’t have to spend a week for our first installation tensorflow. I have been using and contributing too opensource/free software for well over 25 years and tensorflow with GPU is by far the most difficult software to install that I have experienced the last 15 years… Following the official install instruction is just a trap and you get stuck. The only way to get a working install is obviously to go through forums like this one.
@belitskiy I confirm that the issue with
ptxasis resolved with TensorFlow2.17.0.rc1indeed. My test script was executed without errors and smoothly.Running latest CUDA & cuDNN. 2.15.1 finds them, 2.16 does not. And neither does tf-nightly (2.17.x). How can such a large company make such buggy releases?
There should also be instructions for venv users.
On Mon, Apr 8, 2024, 11:48 a.m. Sotiris Gkouzias @.***> wrote:
@SuryanarayanaY Steps to reproduce:
gives this message:
I use rtx3080ti but I think it’s issue for all gpus nvidia-smi:
But installing tensorflow==2.15.1 with python 3.10 works without additional steps
Wanted to drop in and thank everyone for the sleuthing done up to this point. I had no idea about the environmental variable up there that allows you to debug the loading of the libraries so easily. That was neat.
Anyway, to amalgamate all of the suggestions in the thread, you can build an
env_vars.shscript withinenvs/<your environment>/etc/conda/activate.d/in an anaconda-like install that looks like the following:You do indeed need the two
dirnamecommands for TF 2.16. Similarly, you need to put the cuda_nvcc bin directory in the path so ptxas can be found. Doing it this way also generalizes to your specific conda environment. I’ve confirmed that you can perform the intro TF tutorial (https://www.tensorflow.org/tutorials/quickstart/beginner) with this workaround.Having said all that, the TF Linux pip install page badly needs an update. The CUDA Toolkit and CUDNN version numbers are out of date. Since
tensorflow[and-cuda]is now the recommended way to install everything for the GPU, the recommendation to install the other libraries independently is now useless as far as I can tell.Hi Soheil,
I tried your suggestion of having LD_LIBRARY_PATH added, and then install tf.2.16.1. I am happy to report that your suggestion appears to work. I say “appears” only because I tested if tf can detect the GPUs via the command
which returns a non-empty list, so I assume this will work with commands too.
I hope the TF team fixes the issue soon.
Thank you !
Juan E. Vargas
I am of the opinion that just doing RC0 and then final release is not good testing. I hope the 2.16 situation was just a one off situation, to save time (it took the same amount of time for 2.16 with just one RC and final as for older releases with 3 RCs, multiple vulnerability fixes, etc.). I am no longer in the TensorFlow team, just helping here and there on the GitHub issues and PRs.
not it is not, and 2.18.0 does not work either
Thanks @belitskiy for the context. Yes, it looks like this will get resolved for TF 2.17 final release now.
The double registration of those CUDA plugins is brought up in https://github.com/tensorflow/tensorflow/issues/62075, the trial fix for which is what ended up causing this issue, with CUDA/plugins not being detected at all.
The immediate fix was to revert the double-registration fix change, as the lesser of two evils. I pinged someone who may be able to address it properly on that issue
So, there’s no regression and TF 2.17 will have support for GPU.
I’ve been stuck with similar problems the last couple of days. The main problem for me was that TF was able to see the GPU, but I couldn’t get it to actually utilize it. I was getting some ptxas warnings etc. and when I ran the training, it ran on the CPU. I was trying all sorts of tricks that I found on the internet, but nothing solved my problem. Today, I finally tried just running the training in the same container, but for some other (simple) model and it works! So it seems that the only problem was that I had some unsupported operation in my code and I didn’t understand it from the warnings/errors I was receiving. Hope this saves some headaches if anyone gets into a similar situation as I did and ends up here. 😁
I try to use the docker to validate 2.17.0-rc1:
Dockerfile:Run:
Results:
I think this issue has been resolved.
@tilakrayal ,
@alltags additional people. Simply replying here is enough to notify everyone that subscribed, this is not SlackHi,
Tensorflow 2.17.0-rc1 has been released. Could everyone please try to test with the 2.17.0-rc1 and let us know if it is working as expected. https://github.com/tensorflow/tensorflow/releases
Thank you!
A tested workaround to utilize GPU for Linux users:
Create a virtual environment with venv:
python3 -m venv tfActivate the environment
source tf/bin/activateUpgrade pip
pip install --upgrade pipInstall TensorFlow
2.17.0.rc0pip install tensorflow[and-cuda]==2.17.0rc0Create symbolic links to NVIDIA shared libraries
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"I have created a respective pull request still pending review in good faith and for the shake of all users as TensorFlow is “An Open Source Machine Learning Framework for Everyone”.
Can you test the 2.17.0 RC0, please? It is too late to update 2.16, but if 2.17 RC0 doesn’t work, maybe there will be a chance to fix by RC1/final
Everyone that cared about full support of TF is no longer in the team. See above comments for more details and differences
Hooray, it worked, thanks, but I have a mistake with this NUMA. Is this normal? Could it be because I did not install Cuda and Cdn on behalf of the administrator?
Team change, priority change and https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1990133275
@Sivabooshan congrats! However note that:
12.4which is not compatible with TensorFlow version2.16.1and that’s why he might need to install TensorFlow in a virtual environment so as to avoid downgrading CUDA and a potential systemic global pollution (if you install packages globally, they clutter your main Python installation and can potentially interfere with system processes. Virtual environments protect your system-wide environment from this),pip install tensorflow[and-cuda]all required NVIDIA libraries are installed as well. You just need to configure manually the environment variables as appropriate in order to utilize them and run TensorFlow with GPU,2.16.1and utilize GPU locally. That’s why I submitted the pull request in good faith and for the shake of all users as TensorFlow is “An Open Source Machine Learning Framework for Everyone”.Hope that the next patch version of TensorFlow will fix the bug as soon as possible!
Yes, please!
On Fri, Apr 5, 2024 at 12:35 PM Sotiris Gkouzias @.***> wrote:
In colab and gpu VM and docker image, you have cuda cuda installed as a system lib. So tensorflow looks into /usr/lib and finds it. But the standard thing that has worked before 2.16, and the expexpeted thing from pip install tensorflow[and-cuda] is that tensorflow should also look into cuda libraries that were installed via pip. If it looks, it can find nvcc, cufft, cublas,… in there. The problem is that it just doesn’t consider them. Pytorch and tensorflow 2.15 do. You sure can install cuda as a system lib and 2.16 works, but it just would be unnecessary and impossible for users without admin rights. The env-var fix posted above just adds those pip installed cuda libraries to library path so tensorflow finds and uses them.
The fix is simple, you just needs to modify the logic that searches for cuda librariew. but it requires modifying cpp files and recompile tensorflow to be tested, which exceeds resources of most users.
On Fri, Apr 5, 2024, 5:49 a.m. Surya @.***> wrote:
So, @SuryanarayanaY even with NVIDIA GPU driver version 545 still TF 2.16 could not find GPU devices. The bug should be reported and fixed.
It does help! I had the same problem (Ubuntu 22.04, tensorflow 2.16.1): “Cannot dlopen some GPU libraries…”, even though the NVIDIA drivers were properly installed. Tensorflow could not find the libcudnn.so.8 shared library. Thanks to SoheilKhatibi for providing the solution that worked for me. In my case, I found the shared library in my Python virtual environment (named "venv1), under /venv1/lib/python3.10/site-packages/nvidia/cudnn/lib/. So adding a line in .bashrc (“export LD_LIBRARY_PATH=” pointing to the proper path) solved the problem. If you do this, don’t forget to reload .bashr using: source ~/.bashrc
My next problem was that, even though my tensorflow Python code worked fine when run from the terminal, the same script executed within PyCharm still threw the “Cannot dlopen some GPU libraries” error. To solve this, you need to go to PyCharm’s Run menu and select Edit configurations. In the left panel select the script for which you want to solve the GPU error, and open the Environment variables (right panel). Add the user environment variable LD_LIBRARY_PATH and give it the value corresponding to the path you’ve put in .bashr. Save and run, and the error is gone in Pycharm too!
With every year it becomes more and more complicated and difficult to setup tensorflow with nvidia-gpu support on windows. Following worked for me, which is a good example of very bad developer experience.
Install CUDA-Toolkit 12.4 on host machine
Setup Ubuntu 22.04.3 LTS from Windows Store and update it.
Yes try downgrading TensorFlow with
pip install tensorflow==2.15.0.post1. At least this works fine for my Ubuntu20.04.Success!! Ultimately, I located valid ptxas in
...lib/python3.10/site-packages/nvidia/cuda_nvcc/binand manually added the specified path to my environment variables only within the conda virtual environment I created for TensorFlow version 2.16.1 It works like a charm!It does not work unfortunately. It worked with TF<=2.15.1. You would need to test it on non-docker version, since that one actully installs cuda libraries in the container with apt, not pip. Colab is also not a good place to test, because if you do pip list, you can see that cuda libraries are not installed from pypi, but come from some other methods. For this, please test local linux or wsl installation that has only nvidia drivers, not anything else. run pip install tensorflow[and-cuda]==2.16.1in a conda / venv / … environment, and try to list the gpus. It will not work. Then test with 2.15.1 in another environment and see that it works.
@SuryanarayanaY @mihaimaruseac The least that Tensorflow team can do is to test and acknowledge the problem, or say it is not planned, or anything. Just ignoring the problem without even testing is not a supportive act at all.
I am having the same issue.
env: Ubuntu 22.04 + Python 3.10.13 + CUDA 12.4 + tensorflow 2.16.1
Thanks for the workaround, but the path is wrong (conda is missing). Also the file name is arbitrary:
${CONDA_PREFIX}/etc/conda/activate.d/nvidia-lib-dirs.sh. Also the first line can be simplified toand if you don’t have spaces in your environment path, the rest can be simplified to:
The thing is, JAX builds against multiple CUDA versions, whereas TF always pinned to just one version
I’m not sure if this is the root cause, but I resolved my own issue which also surfaced as a “Cannot dlopen some GPU libraries.” error when trying to run
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"To resolve my issue, I followed the tested build versions here: https://www.tensorflow.org/install/source#gpu
and I needed to update my existing installations from cuDNN 9 -> 8.9 and CUDA 12.4->12.3
When you’re on an NVIDIA download page like this one for CUDA Toolkit, don’t just download the latest version. See previous versions by hitting “Archive of Previous CUDA Releases”
@JuanVargas can you try uninstalling your existing CUDA installation to a tested build configuration for TF 2.16 by downgrading to CUDA 12.3?
I followed this post to uninstall my existing cuda installation: https://askubuntu.com/questions/530043/removing-nvidia-cuda-toolkit-and-installing-new-one
@DiegoMont can you try upgrading your cuDNN to 8.9 and CUDA to 12.3?
Solved here by installing the cudnn library:
This issue has been fixed in Tensorflow==2.17.0 version. No need to set any custom environment variables.
Able to access GPU with
tensorflow[and-cuda]==2.17.0If TF 2.17 works then I think this issue can be closed, right?
Thank you for checking
TensorFlow
2.17.0is released ! Tested it right away on the Cats Vs Dogs dataset and YES, it works with GPUs. So, that’s the good news. It turns out that it displays the aforementioned NUMA warning indeed and, in addition, an extra warning:[gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced(the nature of which I am completely unaware). However, noticed that in the certain notebook (used to test the latest version) accuracy was reduced compared to the previous session with TensorFlow version2.16.2(as the last warning warned perhaps).Can you check if the issue with
ptxasis still resolved in 2.17.0rc1?In my environment, tensorflow v2.16.2 is able to use the GPU. At least with the following script, I could confirm that the training works with GPUs.
https://gist.github.com/StephenSzwiec/d4f6f587441f54b9b6c5fed5efd8002d
The ptxas is included in
cuda-nvcc-cu12andtensorflow 2.16.2requirescuda-nvcc-cu12==12.3.107(not 12.3.103).@sgkouzias Perhaps tensorflow is incorrectly reading ptxas 12.3.103, which is installed somewhere in your environment (ex. /usr/local/cuda/bin/ptxas).
@mihaimaruseac, @learning-to-play Yesterday TensorFlow version
2.16.2was released. Tested it today on my PC (with an NVIDIA RTX3060 GPU and Ubuntu 20.04 as my OS). Executed a deep learning model training script but got the following message while training was interrupted:Thus, as I understand it, TF 2.16.2 Fails to work with GPUs as well…
Notes:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"ptxasbug and ultimately enforced TensorFlow2.16.2to work with my GPU:@SeeForTwo @poulsbo Could you please take a look? If there is a fix that needs to be cherry picked to 2.16.2 or 2.17.0, please follow these steps:
@sgkouzias Does this issue happen for both TF 2.16.1 and 2.17.0rc0?
@learning-to-play maybe this can get fixed before final release? TF does not work with GPUs, started failing since TF 2.16 release.
@sgkouzias I checked the official documentation, but I find it not very clear, and seems a bit contradictory: the software requirements state that CUDA and cuDNN should be installed on the machine, but the pip package should install them automatically with Tensorflow right ? Besides, this medium tutorial explain that CUDA should not be installed on Windows side, neither on WSL side, and be installed using the pip package. Maybe I should try to uninstall all CUDA-related on Windows… Concerning your other questions:
I will try to make a clean reinstall of my GPU driver, as well as unistalling CUDA on Windows side. If it doesn’t work, I think it is better to install CUDA and cuDNN manually, along with an oldest TensorFlow version. It is still a shame that the official documentation of such a large and important library is so unclear.
Official documentation has not been updated fot TF 2.16 and still refers to cuda 11.8 not 12.
From: Sotiris Gkouzias @.> Sent: Monday, June 17, 2024 9:54 AM To: tensorflow/tensorflow @.> Cc: Shayan Shahrokhi @.>; Mention @.> Subject: Re: [tensorflow/tensorflow] TF 2.16.1 Fails to work with GPUs (Issue #63362)
Also kindly note that the current issue opened “TF 2.16.1 Fails to work with GPUs” involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.
I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl. Also, for the future, I will say that installing anaconda does not help either. You can install a maximum of 2.10 version on it
@MrOxMasTerhttps://github.com/MrOxMasTer I totally understand your frustration but I reassure you that TensorFlow version 2.16.1 can actually work with your cuda-enabled GPU.
You can try the following:
conda create --name tf python=3.11 conda activate tf
pip install --upgrade pip pip install tensorflow[and-cuda]
Note: This step is required in order to utilize your GPU but not yet included in the official TensorFlow documentation. All NVIDIA libs are installed with TensorFlow due to the fact you ran the command pip install tensorflow[and-cuda] in the previous step!
Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX mkdir -p ./etc/conda/activate.d mkdir -p ./etc/conda/deactivate.d touch ./etc/conda/activate.d/env_vars.sh touch ./etc/conda/deactivate.d/env_vars.sh
Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh
Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH=“${LD_LIBRARY_PATH}”
Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c “import nvidia.cudnn; print(nvidia.cudnn.file)”)))
Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf “%p:”)${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c “import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.file)”)))
Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf “%p:”)${PATH:+:${PATH}}
Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh
Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH=“${ORIGINAL_LD_LIBRARY_PATH}”
Unset environment variables
unset CUDNN_DIR unset PTXAS_DIR
Verify the GPU setup: python3 -c “import tensorflow as tf; print(tf.config.list_physical_devices(‘GPU’))”
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!
Doesn’t work for me 😕 I even reinstalled completely WSL, but I still get an empty list when showing the available devices… Should CUDA be unistalled on Windows side ? When I use “nvidia-smi”, it is written that I have the 12.5 Cuda Version, even if I didn’t install anything on WSL… Is that normal ?
±----------------------------------------------------------------------------------------+ | NVIDIA-SMI 555.52.01 Driver Version: 555.99 CUDA Version: 12.5 | |-----------------------------------------±-----------------------±---------------------+
@ben-jyhttps://github.com/ben-jy frankly I have no clue. Did you check the official documentationhttps://www.tensorflow.org/install/pip#windows-wsl2? Your setup meets the technical requirements? What’s the Python version in WSL2? Is it compatible with TensorFlow 2.16.1? What’s the name of your NVIDIA GPU? The output of the command nvidia-smi in WSL2 seems normal since your GPU driver is installed in Windows. However you could try reinstalling everything (compatible GPU driver, afterwards WSL2 and then TensorFlow)…
— Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2173775825, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRPJAD4P5CMSQLZQOQO5CTZH4BEJAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZTG43TKOBSGU. You are receiving this because you were mentioned.Message ID: @.***>
https://github.com/tensorflow/tensorflow/issues/58629 Also: https://github.com/tensorflow/tensorflow/issues/59918
Google removed the native windows cuda build starting TF 2.11 There is nothing you can do about it, building from source with cuda will also fail in windows.
Thank you for the fast reply. At the moment I’m using the old functions from the keras.src.utils and tf packages, but I’m looking forward to the new release.
@MrOxMasTer I totally understand your frustration but I reassure you that TensorFlow version
2.16.1can actually work with your cuda-enabled GPU.You can try the following:
tfcreated in the previous step run the following commands sequentially:Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIXEnter that directory and create these subdirectories and files:
Edit
./etc/conda/activate.d/env_vars.shas follows:Edit
./etc/conda/deactivate.d/env_vars.shas follows:Verify the GPU setup:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!
Yes, I say for this that you do not need to install graphics drivers in wsl, but installing tensorflow involves not only installing graphics drivers, but also cuDNN and cudaToolkit (possibly TensorRT) and it was not clear to me specifically where it needed to be installed. Then I saw that I needed to install everything inside wsl except the graphics driver
@sgkouzias Thank you. The solution from @niko247 worked and it is what I am using
Hello! I outlined this behavior in a duplicate ticket ( #65842 ). Torch also now installs its CUDA dependencies using the NVIDIA-managed pip packages. However, Torch doesn’t appear to require the LD_LIBRARY_PATH to be set for the linker, like TF still does. I assume this is because they’re manually sourcing libs from the venv. Is this functionality on the roadmap for TF?
Thank you!
Updated the respective pull request (pending review) yesterday. The fix was successfully tested today by @weavermech as well.
Added instructions needed to resolve the ptxas issue.
Updated the respective pull request (pending review) yesterday. The fix was successfully tested today by @weavermech as well.
I thank everyone here for leading me to a simple ‘fix’ for this issue…copy/paste. Sorry, but you guys are leaps and bounds above my understanding of Linux, tensorflow, and wsl2…I just wanted to tinker with a little ML using my rtx 3090. Since Windows apparently isn’t supported, here I am wondering what I’ve gotten myself into. If you are like me and can follow the most basic of instructions, here’s what I did after reading this thread…I used this command TF_CPP_MAX_VLOG_LEVEL=3 python -c “import tensorflow as tf;print(tf.config.list_physical_devices(‘GPU’))” and the output from that told me what libraries weren’t found. More importantly, it told me the one it did find. Because I use wsl2, Windows explorer was all I needed to find the unfindable library files, that were all there in their own nice little folders, including the one that was found. I simply copy/pasted all the files of the ones that weren’t found, into the same location of the one that was found. Worked for me, but one thing’s become abundantly clear…mileage may vary. Good luck. I can’t believe I spent two days trying to figure out all this command line stuff and bash, zsh, echo this and that…how is this even a thing?
Agree. We used to have the instructions for setting the cuda path to the environment variable
LD_LIBRARY_PATHin earlier versions. I think either we need to add these to documentation or atleast has to add a note in pip install guide that user has to setup the path for his own environment.May be same instructions may not work for all environments and hence it might have discorded. Anyways adding a note on same in the installation guide should be must that can avoid confusions.
If anyone from here willing to contribute to add the required notes please feel free to help. The changes can be proposed here at this doc source which will be reflected in this page.
Wanted to stop by with an update. Tried a fresh install this morning via miniforge (anaconda) with python 3.11.8. Followed the instructions on the TF website with the simple
pip install tensorflow[and-cuda]. Same thing – would not recognize the GPU. Adding in theenv_vars.shfix I noted above makes the GPU recognizable.Output from
nvidia-smi:Thanks for all hints. I have written all steps in my blog.
https://mobinshaterian.medium.com/use-gpu-in-tensorflow-on-ubuntu-22-04-f033e59cf5cb
Thanks for the update.
On Sat, Mar 16, 2024, 12:51 James Paul Turner @.***> wrote:
Seriously… I followed the instructions exactly. Yes I used tensorflow[and-cuda]. Under wsl and conda. I’m using vscode and it makes the conda environment in ./.conda Sticking with the previous pip release 2.15.1 then porting to plain pytorch and transformer.
On Fri, 15 Mar 2024, 15:09 Sotiris Gkouzias, @.***> wrote:
I believe that
tensorflow[and-cuda]does not work as expected. For reproduction, I built a simple Docker image:Build and run it:
The output shows the libraries cannot be found, as the same as the above.
Executing
docker run --gpus all -e TF_CPP_MAX_VLOG_LEVEL=3 -e LD_DEBUG=libs test-tf216-cuda python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"will give more information, where I think the search path is entirely wrong. run.logActually you should have version 12.3.107, since it is defined in the extra [and-cuda]. Or did you install differently? Can you check which version you have with
pip show nvidia-cuda-nvcc-cu12and withconda list nvidia-nvcc? You should be able to update.Same issue to me. Is there any official solution for it? I am using WSL2 just today, I don’t want to find and add the PATH manually. I am giving up TF.
Thanks @mihaimaruseac! I had it pinned, also tried with the SHA digest but then realized I had unpinned tensorflow-text in my requirements and that was upgrading TF as well 😛
Again, you are on the tensorflow repo. You closed the wrong issue.
You were asked to close the issue on keras, because it is not a keras issue. This is tensorflow, where proper detection of the cuda libs is definitely still an issue. Please reopen.
I closed the issue because I was asked to do so by a TF/Keras team member, for the reasons that I stated in the closing comment. Just like you (Leigh) I am disappointed, and Idecided not to waste more time. There are strong reasons for some of us (users) to prefer using our own hardware. I hope the TF/keras people get that message. Thanks, Juan
In general, we used to test RC versions before release. For example, we used to have RC0, RC1 and RC2 for TF 2.9. This gave people and downstream teams enough time to test and report issues.
It seems that 2.16.1 only had an RC0 (for 2.16.0).
The release process is (was?) like this:
r2.17)Overall, this process would take
number_of_RCs + 1weeks with a possibility of a few more weeks of delay.However, for 2.16 release, although the branch was cut on Feb 8th, there has been only one RC. Most likely issues can be solved by a patch release