poetry: Can't install detectron2

@malcolmgreaves for completeness this is what you need to keep legacy build.

[build-system]
requires = ["setuptools", "wheel", "torch"]
build-backend = "setuptools.build_meta:__legacy__"

@malcolmgreaves I just tried your workaround to integrate detectron2 with an existing project but still get errors… 😕

My pyproject.toml looks like this:

[tool.poetry.dependencies]
...
detectron2 = { git = "https://github.com/facebookresearch/detectron2.git"}

[build-system]
requires = ["setuptols", "wheel", "torch"]
build-backend = "setuptools.build_meta:__legacy__"

However, when I run poetry install, I get:

PackageInfoError

  Unable to determine package info for path: /tmp/pypoetry-git-detectron20h25u89n
  
  Fallback egg_info generation failed.
  
  Command ['/tmp/tmpi8lubyub/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1, and output: 
  Traceback (most recent call last):
    File "/tmp/pypoetry-git-detectron20h25u89n/setup.py", line 10, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'

  at ~/.poetry/lib/poetry/inspection/info.py:502 in _pep517_metadata
      498│                 try:
      499│                     venv.run("python", "setup.py", "egg_info")
      500│                     return cls.from_metadata(path)
      501│                 except EnvCommandError as fbe:
    → 502│                     raise PackageInfoError(
      503│                         path, "Fallback egg_info generation failed.", fbe
      504│                     )
      505│                 finally:
      506│                     os.chdir(cwd.as_posix())

Any idea why that might be the case?

Wonderful! Thank you a ton 💯 💯 🤗

just using the executable python to run the setup script, like: cd ../detectron2 && /root/.cache/pypoetry/virtualenvs/image-inpainting-WGCBfPov-py3.10/bin/python setup.py install

I managed to get it run by specifying:

[build-system]
requires = ["setuptools", "wheel", "torch==1.10.2+cu113", "torchvision==0.11.3+cu113", "ninja2"]
build-backend = "setuptools.build_meta:__legacy__"

in a detectron2 fork

using the following config

[tool.poetry]
name = "test_detectron_poetry"
version = "0.1.0"
description = ""
authors = ["Benedikt Fuchs <benedikt.fuchs.staw@hotmail.com>"]

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/"
secondary = true
default = false

[tool.poetry.dependencies]
python = "3.8.*"
torch = {version = "1.10.2+cu113", source = "torch"}
torchvision = "0.11.3+cu113"
detectron2 = {git = "https://github.com/helpmefindaname/detectron2.git", rev = "main"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

using poetry==1.2.0b2 following this issue, it is important to mention that this file will not work on the latest non-beta release

and additionally adding https://download.pytorch.org/whl as an extra index in the global git config file

The last step is necessary, as PEP-518 specifies to use isolated builds for building wheels, so the index specified in poetry won’t be used. However, globally configured indexes will also be used by the newly isolated venv.

Some side notes:

• on ubuntu, you need to have gcc and g++ version below 10.*, so you might have to manually downgrade the versions.
• I tried to get it to run on windows first, however there venv didn’t manage to copy the isolated venv to a temp folder, failing the build before any detectron2 related code was executed. I suppose this is not related to this issue. I don’t understand why that happens or if that is specific to my setup or not.
• You can expect to take a very long time for the first time. Even with setting python="3.8.*" it will download wheels for a veriety of python versions. You might want to ensure that pip cache lost is working before starting installing it. I had it disabled for some reason, which made it frustrating when my internet crashed, and I had to start downloading some wheels again.

This seems to no longer work - or at least not in the general case.

The issue that everyone seems to be missing - please correct me if I’m wrong about it - is that simply specifying torch or even torch==1.10.2 in build-system section of detectron2 does not solve the true problem - that is that torch version you install - whatever that version might be - has to be compiled for the same CUDA version you have on your machine.

Since torch available @pypi is compiled by default with CUDA 10.2 (or at least newer versions are), this FORCES CUDA 10.2 onto your system - something the authors of detectron2 have no chance of fixing. @ppwwyyxx

The problem runs even deeper. Even if detectron2 is forked, and proper version of torch is required in the minimal pyproject.toml, there is seems to be no way (@1) of forcing build process to download and use custom wheel - such as available at https://download.pytorch.org/whl/cu113 - which would be required to build detectron2 using specific CUDA version…

^1. please confirm this, I’m new to setuptools and custom package building in general.

@lucinvitae as far as I can tell, the issue here is the package build specification does to convey that it needs torch to successfully build the package. Let me know if this is not the case.

Fro poetry’s perspective, we first need to build the bdist of the package if a compatible one is not already available on the configured indices. This will lead to a poetry attempting to build the package’s wheel, this happens inside an ephemeral isolated build environment according to PEP-517. When creating this environment, the build-system requirements are installed. In this case, since the build scripts need torch available (https://github.com/python-poetry/poetry/issues/2113#issuecomment-711075790). Essentially, something similar to the following is what happens under the hood.

root@c9958409c9fd:/# python -m pip wheel --disable-pip-version-check --use-pep517 git+https://github.com/facebookresearch/detectron2.git
Collecting git+https://github.com/facebookresearch/detectron2.git
  Cloning https://github.com/facebookresearch/detectron2.git to /tmp/pip-req-build-ax2ddphy
  Running command git clone -q https://github.com/facebookresearch/detectron2.git /tmp/pip-req-build-ax2ddphy
  Resolved https://github.com/facebookresearch/detectron2.git to commit c081505af16f54c5a013916b81636d03f7c0a8fd
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmphk82qqi5
       cwd: /tmp/pip-req-build-ax2ddphy
  Complete output (18 lines):
  Traceback (most recent call last):
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-slhwwdsz/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-slhwwdsz/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-slhwwdsz/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 258, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-slhwwdsz/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 10, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'
  ----------------------------------------
WARNING: Discarding git+https://github.com/facebookresearch/detectron2.git. Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmphk82qqi5 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmphk82qqi5 Check the logs for full command output.

However, as @ppwwyyxx identified, there are other limitations that limit the options you have at the moment - ie. you cannot discover automatically the pre-built wheels provided by detectron2 nor can they distribute CUDA wheels on PyPI.

For instance, poetry resolver today does not handle local tags well neither does using link repositories work. This means that you cannot use the indices described in https://github.com/facebookresearch/detectron2/blob/main/INSTALL.md#install-pre-built-detectron2. However, you can use the direct links to the wheels here, but this assumes everwhere you install your package the same wheel can be used (this should be fine for cpu, alternatively you can have various extras. This is being fixed (partially done).

That said, I do not think the python packaging ecosystem has a decent solution for gpu packages anyway - eg. there is no standard way to determine if a cuda enabled wheel can be used in active environment.

• https://discuss.python.org/t/what-to-do-about-gpus-and-the-built-distributions-that-support-them/7125
• https://mail.python.org/archives/list/distutils-sig@python.org/thread/LXLF4YSC4WUZOYRX65DW7CESIX7UUBK5/

Having the dependency metadata specify something like { ..., extra-build-requires = ["torch"] } might be an option, but I am not sure if we should start doing this. On the one hand, this enables poetry users to consume packages where similar issues exist more cleanly (this won’t solve the gpu issue), on the other hand this is a workaround for lack of standard metadata.

Another option is to allow for something like { ..., build-allow-system-site = true }. In this case, the system packages are allowed for package builds.

What poetry should do in the near future are the following:

1. Allow for --find-link type sources in projects.
2. Support local version tags for dependency resolution and package installation.

I still think that the torch dependency need to be declared in the project’s metadata (not in downstream packages) if it is required at build time, so that PEP-517 builds can work correctly.

As for the gpu specific versions, I am not sure where this needs to go. Ideally, there will be a packaging PEP that will help tools like ‘poetry’ to implement a standardised approach.

@joostmeulenbeld I have a workaround here – you can poetry install this as a { git = ... } dependency https://github.com/malcolmgreaves/detectron2

I am going to close this issue as the original issue has been resolved and the current issue has to do with lack of PEP 517 metadata in the package source.

🎉 I have answered my own question – just adding torch to requires in the build-system section is all that’s necessary to get poetry to understand how to build the detectron2 wheel! 🎉

[build-system]
requires = ["setuptools", "wheel", "torch"]

No worries. Wish it was a more helpful response. You can just send them a PR with a minimal pyproject.toml. In the interim, you can use a branch on your fork with the change.