tensorflow: 2.10.0 cuBLAS error
Click to expand!
Issue Type
Build/Install
Source
source
Tensorflow Version
2.10.0
Custom Code
No
OS Platform and Distribution
Ubuntu 7.5.0-3ubuntu1~18.04
Mobile device
No response
Python version
3.8.0
Bazel version
No response
GCC/Compiler version
gcc 7.5.0
CUDA/cuDNN version
NVIDIA-SMI 450.102.04 Driver Version: 450.102.04 CUDA Version: 11.0
GPU model and memory
No response
Current Behaviour?
I use `pip` to install tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`. After installation, when I import tensorflow, there is an error:
`E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered.
I noticed this is an error that occurred before and has an issue closed. But I install the latest 2.10.0 version tf and still meet this error. hope you can help me figure it out. thanks!
Standalone code to reproduce the issue
pip install tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Relevant log output
In [1]: import tensorflow as tf
2022-09-11 03:59:51.293241: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-11 03:59:51.436444: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-09-11 03:59:51.474370: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
</details>
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 14
- Comments: 28 (4 by maintainers)
Commits related to this issue
- Pin TensorFlow to 2.9 to avoid tensorflow/tensorflow#57663 — committed to PythonNut/apptainer-jax by PythonNut 2 years ago
- py-tensorflow: patch for cuBLAS error Version 2.10 produces an error related to cuBLAS: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register ... — committed to glennpj/spack by glennpj 2 years ago
I see the same error with TensorFlow 2.10.0 on Ubuntu 22.04, installed using PIP. Downgrading to 2.9.2 resolves this error. This error can be seen all over TensorFlow documentation site, e.g.: https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras
Don’t close this issue, this is a regression. Should be closed only when 2.11 is released with the fix
This issue is still appearing when I install tensorflow[and-cuda] using poetry. It is installing the 2.14 version.
It is also still appearing in tensorflow 2.15 - had to downgrade to tensorflow 2.9 to get rid of that error.
Has the cherrypick PR merged in r2.10? This error still blocks building from the source with the r2.10 branch.
Hi @donglinz. thanks for the reminder. While I look to check if it has been merged in 2.10, I recommend using the 2.11.0-rc1 for the testing.
Please send a cherrypick PR for
r2.10too to fix the error message. See go/tf-release/cherrypick (internal link)