tensorflow: couldn't install tensorflow with poetry

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Mint 20.1 (based on Ubuntu 20.04)
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No.
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version: 2.4.1
  • Python version: 3.8
  • Installed using virtualenv? pip? conda?: poetry
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: 10.1
  • GPU model and memory: Nvidia RTX 2080

Describe the problem Tensorflow finds the GPU but it isn’t listed under tf.config.list_physical_devices().

Provide the exact sequence of commands / steps that you executed before running into the problem

(base) lukas@Makushin:~$ cd ~/Desktop
(base) lukas@Makushin:~/Desktop$ poetry new poetry-demo
Created package poetry_demo in poetry-demo

At this point I replaced the pyproject.toml file with the following file:

[tool.poetry]
name = "poetry_demo"
version = "0.1.0"
description = ""
authors = ["xxx <xxx@xxx.xx>"]

[tool.poetry.dependencies]
# python
python = "3.8.8"
# data science
numpy = "*"
pandas = "*"
geopandas = "*"
matplotlib = "*"
sklearn = "*"
tensorflow = "^2.4.1"
# image processing
Pillow = "*"
rasterio = "*"


(base) lukas@Makushin:~/Desktop$ cd poetry-demo
(base) lukas@Makushin:~/Desktop/poetry-demo$ poetry install
Creating virtualenv poetry-demo-bLNYl46S-py3.8 in /home/lukas/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (6.6s)

Writing lock file

Package operations: 86 installs, 0 updates, 0 removals

  • Installing certifi (2020.12.5)
  • Installing chardet (4.0.0)
  • Installing idna (2.10)
... 
  • Installing pyasn1 (0.4.8)
  • Installing urllib3 (1.26.4)
  • Installing cachetools (4.2.2)
  • Installing oauthlib (3.1.0)
  • Installing pyasn1-modules (0.2.8)
  • Installing requests (2.25.1)
  • Installing rsa (4.7.2)
  • Installing six (1.15.0)
  • Installing click (7.1.2)
  • Installing google-auth (1.30.0)
  • Installing numpy (1.19.5)
  • Installing requests-oauthlib (1.3.0)
  • Installing absl-py (0.12.0)
  • Installing attrs (21.1.0)
  • Installing click-plugins (1.1.1)
  • Installing cligj (0.7.1)
  • Installing google-auth-oauthlib (0.4.4)
  • Installing grpcio (1.32.0)
  • Installing joblib (1.0.1)
  • Installing markdown (3.3.4)
  • Installing markupsafe (1.1.1)
  • Installing munch (2.5.0)
  • Installing protobuf (3.15.8)
  • Installing pyparsing (2.4.7)
  • Installing python-dateutil (2.8.1)
  • Installing pytz (2021.1)
  • Installing scipy (1.6.3)
  • Installing tensorboard-data-server (0.6.1)
  • Installing tensorboard-plugin-wit (1.8.0)
  • Installing threadpoolctl (2.1.0)
  • Installing werkzeug (1.0.1)
  • Installing affine (2.3.0)
  • Installing alabaster (0.7.12)
  • Installing appdirs (1.4.4)
  • Installing astunparse (1.6.3)
  • Installing babel (2.9.1): Installing...
  • Installing cycler (0.10.0)
  • Installing docutils (0.16): Installing...
  • Installing fiona (1.8.19): Installing...
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0): Installing...
  • Installing imagesize (1.2.0)
  • Installing imagesize (1.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing fiona (1.8.19)
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing cycler (0.10.0)
  • Installing docutils (0.16): Installing...
  • Installing fiona (1.8.19)
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing babel (2.9.1)
  • Installing cycler (0.10.0)
  • Installing docutils (0.16): Installing...
  • Installing fiona (1.8.19)
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing fiona (1.8.19)
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing docutils (0.16)
  • Installing fiona (1.8.19)
  • Installing flatbuffers (1.12)
  • Installing gast (0.3.3)
  • Installing google-pasta (0.2.0)
  • Installing h5py (2.10.0)
  • Installing imagesize (1.2.0)
  • Installing iniconfig (1.1.1)
  • Installing jinja2 (2.11.3)
  • Installing keras-preprocessing (1.1.2)
  • Installing kiwisolver (1.3.1)
  • Installing mypy-extensions (0.4.3)
  • Installing opt-einsum (3.3.0)
  • Installing packaging (20.9)
  • Installing pandas (1.2.4): Installing...
  • Installing pandas (1.2.4)
  • Installing pathspec (0.8.1)
  • Installing pillow (8.2.0)
  • Installing pluggy (0.13.1)
  • Installing py (1.10.0)
  • Installing pygments (2.9.0)
  • Installing pyproj (3.0.1)
  • Installing regex (2021.4.4)
  • Installing scikit-learn (0.24.2)
  • Installing snowballstemmer (2.1.0)
  • Installing shapely (1.7.1)
  • Installing snuggs (1.4.7)
  • Installing sphinxcontrib-applehelp (1.0.2)
  • Installing sphinxcontrib-devhelp (1.0.2)
  • Installing sphinxcontrib-htmlhelp (1.0.3)
  • Installing sphinxcontrib-jsmath (1.0.1)
  • Installing sphinxcontrib-qthelp (1.0.3)
  • Installing tensorboard (2.5.0)
  • Installing tensorflow-estimator (2.4.0)
  • Installing sphinxcontrib-serializinghtml (1.1.4)
  • Installing termcolor (1.1.0)
  • Installing toml (0.10.2)
  • Installing typing-extensions (3.7.4.3)
  • Installing wrapt (1.12.1)
  • Installing black (21.5b0)
  • Installing geopandas (0.9.0)
  • Installing matplotlib (3.4.1)
  • Installing pytest (6.2.4)
  • Installing rasterio (1.2.3)

  • Installing sklearn (0.0)
  • Installing sphinx (3.5.4)
  • Installing tensorflow (2.4.1)

Installing the current project: poetry_demo (0.1.0)
(base) lukas@Makushin:~/Desktop/poetry-demo$ poetry run python
Python 3.8.8 (default, Apr 13 2021, 19:58:26) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-05-06 17:51:27.535624: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-06 17:51:27.535643: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
>>> tf.config.list_physical_devices()
2021-05-06 17:51:57.675020: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-06 17:51:57.675844: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-05-06 17:51:57.701899: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-06 17:51:57.702606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:2d:00.0 name: NVIDIA GeForce RTX 2080 computeCapability: 7.5
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 7.77GiB deviceMemoryBandwidth: 417.23GiB/s
2021-05-06 17:51:57.702710: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-06 17:51:57.702793: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2021-05-06 17:51:57.702866: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2021-05-06 17:51:57.704592: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-05-06 17:51:57.704887: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-05-06 17:51:57.706566: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-05-06 17:51:57.706678: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2021-05-06 17:51:57.706759: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2021-05-06 17:51:57.706778: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]
>>>

Any other info / logs Another way I tried was to install Tensorflow into a fresh Anaconda environment using conda create --name tf_gpu tensorflow-gpu. That way the GPU was listed in tf.config.list_physical_devices(). However I would like to avoid Anaconda if posibble because it has other, unrelated drawbacks.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

This issue should be reopened

+17
jonathansp on Jun 24, 2022

This issue should be reopened

This is not resolved. #58770 did not fix it. I can reproduce the issue exactly as described in my OP with the changed version numbers:

  • Poetry 1.4.2 (fresh install)
  • Python 3.10.10 (changed in pyproject.toml)
  • Tensorflow 2.12.0 - newest release (automatically installed as per tensorflow = "^2.4.1" in pyproject.toml)
  • CUDA 11.7 (automatically installed)

OS and GPU are the same as above.

+3
joooeey on May 4, 2023

Yes this works for me now as well

+2
thoughtfuldata on May 4, 2023

I dont even know what OP did to solve this according to:

I solved this by combining environment management from Anaconda and Poetry like so:

poetry update
poetry run conda create -y --name tf_gpu tensorflow-gpu
poetry run conda activate tf_gpu
poetry run python

Then the GPU was recognized.

to run cuda from poetry I need to install it with poetry, and when I do that I get errors that I cant use the cuda executable installed that way… like that.

No, I didn’t manually install cuda, and I didn’t install it from poetry. All I did was install tf_gpu with poetry run conda create -y --name tf_gpu tensorflow-gpu (2nd line of the code block). This installs tf_gpu with conda from within poetry.

This worked on the system described above. Unfortunately, I have no record of what poetry version I was using but it was a recent version in May 2021.

+1
joooeey on Apr 27, 2023
Read more comments on GitHub
← tensorflow: Couldn't find field google.protobuf.EnumDescriptorProto.EnumReservedRange.start
tensorflow: CPU Build Fails on OSX Sierra  →