pex: Packages at relative paths not found

I have a monorepo and am using Poetry to manage all the individual packages. (I’m not using Pants for this for other reasons.)

Packages depend on a mix of regular external libraries in pypi and local packages. For example:

[tool.poetry]
name = "ns_lexio_db"
version = "0.1.0"
description = ""
authors = []
license = "Proprietary"

[tool.poetry.dependencies]
python = "^3.6.5"
alembic = "0.9.7"
Click = "7.0"
ns_database_seeding = { path = "../lib/database_seeding", develop = true }
ns_python_core = { path = "../lib/python_core", develop = true }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
lexio-db = "ns_lexio_db.cli:cli"

From here, the ns_database_seeding package depends on other internal packages and use path links too.

Next I export the requirements:

poetry export -f requirements.txt --without-hashes -o requirements.txt

which gives something like:

...
newrelic==5.24.0.153; python_full_version >= "3.6.5" and python_full_version < "4.0.0"
...
ns-feature-flag-client @ file:///Users/jdrake/talos2/lib/feature_flag_client; python_full_version >= "3.6.5" and python_full_version < "4.0.0"
...

To build the pex I run:

pex --requirement=requirements.txt --script=lexio-db --sources-directory=. --output-file=lexio-db.pex  

But that fails with:

    ERROR: Command errored out with exit status 1:
     command: /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/bin/python3.6 /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/tmpfuc9lsur
         cwd: /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-download-a3hcfey8/ns-admin
    Complete output (16 lines):
    Traceback (most recent call last):
      File "/Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
        main()
      File "/Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/masonry/api.py", line 43, in prepare_metadata_for_build_wheel
        poetry = Factory().create_poetry(Path(".").resolve(), with_dev=False)
      File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/factory.py", line 93, in create_poetry
        self.create_dependency(name, constraint, root_dir=package.root_dir)
      File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/factory.py", line 251, in create_dependency
        extras=constraint.get("extras", []),
      File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/packages/directory_dependency.py", line 41, in __init__
        raise ValueError("Directory {} does not exist".format(self._path))
    ValueError: Directory ../feature_flag_client does not exist
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/bin/python3.6 /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/tmpfuc9lsur Check the logs for full command output.
pid 80108 -> /Users/jdrake/.pex/venvs/8fa46d08dd05e0160395f355cccb51e48f384a1b/30f8c74d3988c60c0f5aeae7604ec76b6ce2368a/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /Users/jdrake/.pex download --dest /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/tmpanbshq0u/Users.jdrake..pyenv.versions.3.6.8.bin.python3.6 --requirement requirements.txt --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None

It seems like pex (or pip under the hood) is picking up on the pyproject.toml files for internal dependencies (via pip hook) but misinterpreting the relative paths. They are in each pyproject.toml file and poetry.lock file relative to that project, but it seems like pex or pip is interpreting them as relative to some virtualenv or current directory or something.

I am using the newest versions of pip, poetry, and pex.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (12 by maintainers)

Most upvoted comments

Ok, @jdrake the only way to get this working sanely in Pex will be if #1361 gets implemented; otherwise, Poetry is breaking too many PEPs to work with without writing a bunch of special case code to deal with Poetry projects. I’m going to close as won’t fix in favor of #1361. Please speak up though if you have ideas on other ways for Pex to handle this.