tensorflow: 2.10.0 cuBLAS error

Click to expand!

Issue Type

Build/Install

Source

source

Tensorflow Version

2.10.0

Custom Code

No

OS Platform and Distribution

Ubuntu 7.5.0-3ubuntu1~18.04

Mobile device

No response

Python version

3.8.0

Bazel version

No response

GCC/Compiler version

gcc 7.5.0

CUDA/cuDNN version

NVIDIA-SMI 450.102.04 Driver Version: 450.102.04 CUDA Version: 11.0

GPU model and memory

No response

Current Behaviour?

I use `pip` to install tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`. After installation, when I import tensorflow, there is an error:
`E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered.
I noticed this is an error that occurred before and has an issue closed. But I install the latest 2.10.0 version tf and still meet this error. hope you can help me figure it out. thanks!

Standalone code to reproduce the issue

pip install tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Relevant log output

In [1]: import tensorflow as tf
2022-09-11 03:59:51.293241: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-11 03:59:51.436444: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-09-11 03:59:51.474370: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
</details>

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 14
  • Comments: 28 (4 by maintainers)

Commits related to this issue

Most upvoted comments

I see the same error with TensorFlow 2.10.0 on Ubuntu 22.04, installed using PIP. Downgrading to 2.9.2 resolves this error. This error can be seen all over TensorFlow documentation site, e.g.: https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras

Screen Shot 2022-09-12 at 12 57 59 PM

Don’t close this issue, this is a regression. Should be closed only when 2.11 is released with the fix

This issue is still appearing when I install tensorflow[and-cuda] using poetry. It is installing the 2.14 version.

This issue is still appearing when I install tensorflow[and-cuda] using poetry. It is installing the 2.14 version.

It is also still appearing in tensorflow 2.15 - had to downgrade to tensorflow 2.9 to get rid of that error.

Has the cherrypick PR merged in r2.10? This error still blocks building from the source with the r2.10 branch.

Hi @donglinz. thanks for the reminder. While I look to check if it has been merged in 2.10, I recommend using the 2.11.0-rc1 for the testing.

@rishikasinha-tf, would it be possible to include the PR in a TF 2.10.1 release? The log message is confusing as it (incorrectly) implies that an error has occurred that would prevent TF from running correctly.

Please send a cherrypick PR for r2.10 too to fix the error message. See go/tf-release/cherrypick (internal link)