pytorch_sparse: Error in setup.py "No module named 'torch'" when installing with Poetry
When I try to install torch-sparse using Poetry, I’m getting the following error which occurs in setup.py:
ModuleNotFoundError: No module named 'torch'
The reason is that torch-sparse imports torch in setup.py while torch is not yet installed. Since those torch imports are only needed to build compiled extensions, it should be possible to avoid importing torch when installing the torch-sparse wheel package.
These are commands to reproduce the problem (tested using Poetry v1.1.7):
$ poetry init -n --python '^3.6.2' --dependency torch --dependency torch-sparse
$ poetry install
Creating virtualenv torch-sparse-poetry in /tmp/torch-sparse-poetry/.venv
Updating dependencies
Resolving dependencies... (77.1s)
Writing lock file
Package operations: 6 installs, 0 updates, 0 removals
• Installing numpy (1.19.5)
• Installing dataclasses (0.8)
• Installing scipy (1.5.4)
• Installing typing-extensions (3.10.0.0)
• Installing torch (1.9.0)
• Installing torch-sparse (0.6.11): Failed
EnvCommandError
Command ['/tmp/torch-sparse-poetry/.venv/bin/pip', 'install', '--no-deps', '$HOME/.cache/pypoetry/artifacts/59/cf/7b/23094d3d3aa79d571458529d8031882ce27d36db73083987acdab34868/torch_sparse-0.6.11.tar.gz'] errored with the following return code 1, and output:
Processing $HOME/.cache/pypoetry/artifacts/59/cf/7b/23094d3d3aa79d571458529d8031882ce27d36db73083987acdab34868/torch_sparse-0.6.11.tar.gz
ERROR: Command errored out with exit status 1:
command: /tmp/torch-sparse-poetry/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-vk1oqsni/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-vk1oqsni/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-2tsa72d0
cwd: /tmp/pip-req-build-vk1oqsni/
Complete output (5 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-vk1oqsni/setup.py", line 8, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
----------------------------------------
WARNING: Discarding file://$HOME/.cache/pypoetry/artifacts/59/cf/7b/23094d3d3aa79d571458529d8031882ce27d36db73083987acdab34868/torch_sparse-0.6.11.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 21.1.3; however, version 21.2.2 is available.
You should consider upgrading via the '/tmp/torch-sparse-poetry/.venv/bin/python -m pip install --upgrade pip' command.
at ~/.local/share/pypoetry/venv/lib/python3.6/site-packages/poetry/utils/env.py:1101 in _run
1097│ output = subprocess.check_output(
1098│ cmd, stderr=subprocess.STDOUT, **kwargs
1099│ )
1100│ except CalledProcessError as e:
→ 1101│ raise EnvCommandError(e, input=input_)
1102│
1103│ return decode(output)
1104│
1105│ def execute(self, bin, *args, **kwargs):
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 19
- Comments: 18 (3 by maintainers)
Commits related to this issue
- Generic `kumo_loader` function that can points to snowflake/s3/local dataset (#156) * Add a generic `kumo_loader` function that can points to snowflake/s3/local dataset * Clean up code, switch tes... — committed to RexYing/pytorch_sparse by JiaxuanYou 5 years ago
- fix: torch import error while installing torch_sparse with pip refer to https://github.com/rusty1s/pytorch_sparse/issues/156#issuecomment-1304869772 for details — committed to WenjieDu/PyPOTS by WenjieDu 4 years ago
I got around the issue in the following way:
(1) Configure in the .toml file the source URL where the wheels are pulled from:
poetry source add torch-wheels https://data.pyg.org/whl/torch-1.12.0+cpu.htmlCheck that the parts
torch-1.12.0andcpuin the url match yourtorchand cuda/cpu versions. Please refer here for different options.Running the command should create the following kind of section to your .toml file.
(2) Add and install the
torch_sparsepackage (along with other necessary packages) using the source configured in the .toml file:poetry add --source torch-wheels pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_convNow you should see this in the .toml file
…and everything should work fine.
Clearly this is not a scalable solution if the repo is used by several different people with different cpu/cuda setups, but works as a temporary workaround.
@abrahme no worries, your response seems like the right way to go!
Yeah, I’d guess that’s what others do. ie: the
[tool.poetry.dependencies]would contain most deps but omittorch,torch-sparse, etc. Then, install those separately before or after thepoetry install.Another approach I’ve seen people do is hard code the URLs to pull wheels/etc from. You can specify markers so the right file is used for each OS/CPU, but you would have to just hard code the cuda version (eg:
cu116) in the URL and ensure it aligns with your host(s).Thanks for the clarification. I ended up just using
pyenvinstead and abandoning poetryWe currently have an install script that installs torch and then these packages. After that, we run
poetry install. Since the installed versions of torch* don’t match what poetry has locked (poetry expects eg:X.X.X, but seesX.X.X+cu116or whatever) and would try to reinstall them, we have some hacky code that renames the installed packages (insite-packages) to remove the+cuXYZfrom the folder/metadata so it matches poetry’s expectations.TL;DR pretty hacky. 😅 I think others may just avoid placing torch* in their
pyproject.toml(assuming they don’t have any transitive deps with it)?On Mac, with @hemmokarja solution to setup a secondary source failed
with
poetry add pyg-lib --source torch-wheels -vraisingUnable to find installation candidates for pyg-lib (0.3.1+pt20cpu)After hours of frustration, I realised by checking in the source list, that poetry was trying to fetch non-macos versions (ending with “cpu”: 0.3.1+pt21cpu)
Simply running
poetry add pyg-lib="0.3.0+pt21" --source torch-wheels -vworked. I’m sure it generalizes to torch_{sparce, scatter, …} too. Hope it helps !