poetry: Dependency resolution problem with non-pypi repository

Version: Poetry 0.12.10

I have a private PyPI server for company internal packages. Something in poetry seems to get confused when that private package has public dependencies. Specifically this case:

private-package depends on public-package which itself has further (public) dependencies.

poetry add private-package
poetry add public-package fails with [PackageNotFound]: Package [<private-package>] not found.

However poetry add works for public packages not depended on by private-package and also works for public packages which have no dependencies on their own, even if depended on by private-package.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 20 (11 by maintainers)

Most upvoted comments

Any news on this? Is there a good argument not to use --extra-index-url rather than --index-url?

I’ve actually thought about a good reason not to use --extra-index-url. Say you have a private package privpkg on your private repo. If someone now uploads privpkg to the public repo, using --extra-index-url would pull that in.

Maybe a better solution would be something like:

[tool.poetry.dependencies]
privpkg = { version = "^2.13.0", repo = "privaterepo" }

Though this raises the question how private and public dependencies of privpkg are handled, I guess the lockfile could take care of that?

ojii on Feb 6, 2019

Is there a way to make private repositories behave like --extra-index-url rather than --index-url (in pip)? Ideally I’d like poetry to try pypi.org first, then fall back to the company pypi.

ojii on Dec 11, 2018

Any update on this. I just spend quite some time on migrating the whole project to poetry and now i am stuck at this point.

scottyHH on Feb 6, 2019

@sdispater After some debugging, I figured out the issue. It mostly on my end. The URL I used to specify my index is the same as I use for my pip configuration (pip.conf). However, it does not carry the /+simple path that poetry needed. Apparently, native pip adds that on automatically when it is contacting an index, but poetry needed it expicitly.

Once I ammended my pyproject.toml file accordingly, it worked great.

Thanks again for the great tool!

working pyproject.toml:

[tool.poetry]
name = "pa-mockup"
version = "0.1.0"
description = ""
authors = ["Tucker Beck <tucker.beck@---.com>"]

[tool.poetry.dependencies]
python = "^3.7"
pendulum = "^2.0"
flask-apispec = "^0.8.0"
flask = "^1.0"
flask-buzz = "^0.1.14"
flask-jwt-extended = "^3.18"
flask-cors = "^3.0"
flask-mail = "^0.9.1"
flask-redis = "^0.3.0"
fetch-config = "^0.1.3"

[tool.poetry.dev-dependencies]
pytest = "^3.0"
ipdb = "^0.12.0"

[[tool.poetry.source]]
name = "pa-devpi"
url = "http://---.---.--.--:3141/pa-team/pa/+simple"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

dusktreader on May 7, 2019

Any news on this? Is there a good argument not to use --extra-index-url rather than --index-url?

cipriantarta on Jan 14, 2019

Here’s the -vvvvv log (company-pypi is a private pypi server, private-package depends on aiohttp):

$ poetry add -vvvvv  aiohttp
Using virtualenv: /Users/jonas.obrist/Library/Caches/pypoetry/virtualenvs/repro-py3.7
company-pypi: 146 packages found for aiohttp *
Using version ^3.4 for aiohttp

Updating dependencies
Resolving dependencies...
   1: fact: repro is 0.1.0
   1: derived: repro
   1: fact: repro depends on private-package (^0.0.11)
   1: fact: repro depends on attrs (^18.2)
   1: fact: repro depends on requests (^2.20)
   1: fact: repro depends on aiohttp (^3.4)
   1: selecting repro (0.1.0)
   1: derived: aiohttp (^3.4)
   1: derived: requests (^2.20)
   1: derived: attrs (^18.2)
   1: derived: private-package (^0.0.11)
company-pypi: 5 packages found for aiohttp >=3.4,<4.0
company-pypi: The cache for aiohttp 3.4.4 is outdated. Refreshing.
company-pypi: Downloading sdist: aiohttp-3.4.4.tar.gz
   1: fact: aiohttp (3.4.4) depends on attrs (>=17.3.0)
   1: fact: aiohttp (3.4.4) depends on chardet (>=2.0,<4.0)
   1: fact: aiohttp (3.4.4) depends on multidict (>=4.0,<5.0)
   1: fact: aiohttp (3.4.4) depends on async_timeout (>=3.0,<4.0)
   1: fact: aiohttp (3.4.4) depends on yarl (>=1.0,<2.0)
   1: selecting aiohttp (3.4.4)
   1: derived: yarl (>=1.0,<2.0)
   1: derived: async_timeout (>=3.0,<4.0)
   1: derived: multidict (>=4.0,<5.0)
   1: derived: chardet (>=2.0,<4.0)
company-pypi: The cache for requests 2.20.1 is outdated. Refreshing.
company-pypi: Downloading wheel: requests-2.20.1-py2.py3-none-any.whl
   1: fact: requests (2.20.1) depends on chardet (>=3.0.2,<3.1.0)
   1: fact: requests (2.20.1) depends on idna (>=2.5,<2.8)
   1: fact: requests (2.20.1) depends on urllib3 (>=1.21.1,<1.25)
   1: fact: requests (2.20.1) depends on certifi (>=2017.4.17)
   1: selecting requests (2.20.1)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<1.25)
   1: derived: idna (>=2.5,<2.8)
   1: derived: chardet (>=3.0.2,<3.1.0)
company-pypi: The cache for attrs 18.2.0 is outdated. Refreshing.
company-pypi: Downloading wheel: attrs-18.2.0-py2.py3-none-any.whl
   1: selecting attrs (18.2.0)
company-pypi: The cache for private-package 0.0.11 is outdated. Refreshing.
company-pypi: Downloading wheel: private-package-0.0.11-py3-none-any.whl
   1: fact: private-package (0.0.11) depends on attrs (*)
   1: fact: private-package (0.0.11) depends on aiobotocore (*)
   1: fact: private-package (0.0.11) depends on boto3 (<=1.5)
   1: fact: private-package (0.0.11) depends on aiohttp (>3,<=3.0.9)
   1: derived: not private-package (0.0.11)
company-pypi: 0 packages found for private-package >0.0.11,<0.0.12
   1: Version solving took 8.155 seconds.
   1: Tried 1 solutions.
                                  
[PackageNotFound]  
Package [private-package] not found.           
                                  
Exception trace:
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/jonas.obrist/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/jonas.obrist/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/jonas.obrist/.poetry/lib/poetry/console/commands/add.py in handle() at line 139
   status = installer.run()
 /Users/jonas.obrist/.poetry/lib/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /Users/jonas.obrist/.poetry/lib/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /Users/jonas.obrist/.poetry/lib/poetry/puzzle/solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 /Users/jonas.obrist/.poetry/lib/poetry/puzzle/solver.py in _solve() at line 171
   self._package, self._provider, locked=locked, use_latest=use_latest
 /Users/jonas.obrist/.poetry/lib/poetry/mixology/__init__.py in resolve_version() at line 7
   return solver.solve()
 /Users/jonas.obrist/.poetry/lib/poetry/mixology/version_solver.py in solve() at line 79
   next = self._choose_package_version()
 /Users/jonas.obrist/.poetry/lib/poetry/mixology/version_solver.py in _choose_package_version() at line 354
   packages = self._provider.search_for(dependency)
 /Users/jonas.obrist/.poetry/lib/poetry/puzzle/provider.py in search_for() at line 146
   allow_prereleases=dependency.allows_prereleases(),
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pool.py in find_packages() at line 65
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in find_packages() at line 104
   info = self.get_package_info(name)
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in get_package_info() at line 228
   name, lambda: self._get_package_info(name)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cachy/repository.py in remember_forever() at line 174
   val = value(callback)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cachy/helpers.py in value() at line 6
   return val()
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in <lambda>() at line 228
   name, lambda: self._get_package_info(name)
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in _get_package_info() at line 234
   raise PackageNotFound("Package [{}] not found.".format(name))

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

ojii on Dec 6, 2018