pex: Packages at relative paths not found
I have a monorepo and am using Poetry to manage all the individual packages. (I’m not using Pants for this for other reasons.)
Packages depend on a mix of regular external libraries in pypi and local packages. For example:
[tool.poetry]
name = "ns_lexio_db"
version = "0.1.0"
description = ""
authors = []
license = "Proprietary"
[tool.poetry.dependencies]
python = "^3.6.5"
alembic = "0.9.7"
Click = "7.0"
ns_database_seeding = { path = "../lib/database_seeding", develop = true }
ns_python_core = { path = "../lib/python_core", develop = true }
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.scripts]
lexio-db = "ns_lexio_db.cli:cli"
From here, the ns_database_seeding package depends on other internal packages and use path links too.
Next I export the requirements:
poetry export -f requirements.txt --without-hashes -o requirements.txt
which gives something like:
...
newrelic==5.24.0.153; python_full_version >= "3.6.5" and python_full_version < "4.0.0"
...
ns-feature-flag-client @ file:///Users/jdrake/talos2/lib/feature_flag_client; python_full_version >= "3.6.5" and python_full_version < "4.0.0"
...
To build the pex I run:
pex --requirement=requirements.txt --script=lexio-db --sources-directory=. --output-file=lexio-db.pex
But that fails with:
ERROR: Command errored out with exit status 1:
command: /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/bin/python3.6 /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/tmpfuc9lsur
cwd: /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-download-a3hcfey8/ns-admin
Complete output (16 lines):
Traceback (most recent call last):
File "/Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
main()
File "/Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/masonry/api.py", line 43, in prepare_metadata_for_build_wheel
poetry = Factory().create_poetry(Path(".").resolve(), with_dev=False)
File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/factory.py", line 93, in create_poetry
self.create_dependency(name, constraint, root_dir=package.root_dir)
File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/factory.py", line 251, in create_dependency
extras=constraint.get("extras", []),
File "/private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/pip-build-env-wq5hwh1m/overlay/lib/python3.6/site-packages/poetry/core/packages/directory_dependency.py", line 41, in __init__
raise ValueError("Directory {} does not exist".format(self._path))
ValueError: Directory ../feature_flag_client does not exist
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/bin/python3.6 /Users/jdrake/.pex/venvs/s/2f33bdcc/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/tmpfuc9lsur Check the logs for full command output.
pid 80108 -> /Users/jdrake/.pex/venvs/8fa46d08dd05e0160395f355cccb51e48f384a1b/30f8c74d3988c60c0f5aeae7604ec76b6ce2368a/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /Users/jdrake/.pex download --dest /private/var/folders/4l/fzzb6qx11z34_6xfrs9w0pp80000gp/T/tmpanbshq0u/Users.jdrake..pyenv.versions.3.6.8.bin.python3.6 --requirement requirements.txt --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None
It seems like pex (or pip under the hood) is picking up on the pyproject.toml files for internal dependencies (via pip hook) but misinterpreting the relative paths. They are in each pyproject.toml file and poetry.lock file relative to that project, but it seems like pex or pip is interpreting them as relative to some virtualenv or current directory or something.
I am using the newest versions of pip, poetry, and pex.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (12 by maintainers)
Ok, @jdrake the only way to get this working sanely in Pex will be if #1361 gets implemented; otherwise, Poetry is breaking too many PEPs to work with without writing a bunch of special case code to deal with Poetry projects. I’m going to close as won’t fix in favor of #1361. Please speak up though if you have ideas on other ways for Pex to handle this.