poetry: Project with path dependency cannot be installed with `pip install .` (or via requirements file) due to "RequirementParseError: Invalid URL"

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS 11.4
Poetry version: 1.1.13, 1.2.0b2.dev0 (f2c365fb6a23e78205681d6aa4111c888e9ee6ab)
Link of a Gist with the contents of your pyproject.toml file: https://github.com/huonw/poetry-path-deps-issue

Issue

We’re working within a monorepo where we have several projects that depend on each other via path = "..." relative dependencies. Installing such a project directly with pip install fails with pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a.

Why are we trying to do this? We want to have access to pip install --target to install packages to a specific directory, when packaging for an AWS Lambda .zip file. This means we’re exporting to a requirements file and installing that.

This is potentially caused by only absolute paths being turned into a URL, in https://github.com/python-poetry/poetry-core/blob/c5f4cda45402c38bb87ab73d0e32de509ba68f41/src/poetry/core/packages/directory_dependency.py#L125

Potentially related issues:

similar error messages: python-poetry/poetry#3930, python-poetry/poetry#3899, python-poetry/poetry#4709, python-poetry/poetry#4868
similar set-up: python-poetry/poetry-plugin-export#147
python-poetry/poetry#936 for the monorepo style with a lot of path deps

Example

https://github.com/huonw/poetry-path-deps-issue contains a full reproducible example:

cd /tmp
git clone https://github.com/huonw/poetry-path-deps-issue.git
pip install poetry-path-deps-issue/b

The core layout is:

.
├── a
│   ├── a
│   │   └── __init__.py
│   └── pyproject.toml
├── b
│   ├── b
│   │   └── __init__.py
│   └── pyproject.toml
└── c
    ├── c
    │   └── __init__.py
    ├── poetry.lock
    └── pyproject.toml

Where b/pyproject.toml contains a dependency a = { path = "../a" } (c is left over from an older version of this issue, see below).

The pip install . fails with:

Processing /poetry-path-deps-issue/b
...
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a
...

Click for older details using poetry export

c/pyproject.toml contains a dependency b = { path = "../b" }.

Commands:

cd poetry-path-deps-issue/c
poetry export -o requirements.txt
pip install -r requirements.txt

After running those commands, the requirements.txt contains something like:

a @ file:///private/tmp/poetry-path-deps-issue/a; python_version >= "3.9" and python_version < "4.0"
b @ file:///private/tmp/poetry-path-deps-issue/b; python_version >= "3.9" and python_version < "4.0"

And the pip install output is:

Processing /private/tmp/poetry-path-deps/a
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Processing /private/tmp/poetry-path-deps/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
...
  File ".../lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

Workaround

We’re preprocessing all pyproject.toml files to remove the path dependencies, after exporting the requirements, but before installing them (via find -name pyproject.toml | xargs sed -i '/path =/d'). The export should contain all of the dependencies required, so we don’t them to be re-read out of each pyproject.toml.

Full output

Processing /poetry-path-deps-issue/b
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 117, in __init__
    raise InvalidRequirement(f"Invalid URL: {req.url}")
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: ../a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 341, in run
    requirement_set = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 237, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 246, in iter_dependencies
    requires = self.dist.iter_dependencies() if with_requires else ()
  File "/usr/local/lib/python3.9/site-packages/pip/_internal/metadata/pkg_resources.py", line 200, in iter_dependencies
    return self._dist.requires(extras)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: ../a

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 8
Comments: 27 (18 by maintainers)

Commits related to this issue

Use file:// urls for path and directory dependencies (#512) fixes https://github.com/python-poetry/poetry/issues/5273 — committed to python-poetry/poetry-core by dimbleby 2 years ago

Most upvoted comments

Just got same issue, i was able to install correctly with:

# relative deps from pyproject.toml are converted to absolute path
poetry export -f requirements.txt -o requirements.txt --without-hashes
pip install -r requirements.txt
# ignore dependencies as it has already been installed by requirements
pip install . --no-dependencies

jcosmao on Jul 23, 2022

yep, the discussion here has drifted significantly but I think that what this issue about is just that poetry-core puts relative paths in metadata, and this is easily fixed by anyone who cares: tweak this code to use an absolute version of the full path that is already known to the directory dependency.

(and similar for file dependency)

dimbleby on Oct 11, 2022

I get the feeling that most folks using path dependencies are working with some sort of monorepo setup. There are two use cases I’ve seen for path dependencies in these workflows:

Include the source code and dependencies
Treat it like a published package with a version

For the first case I think the logical thing to do is to treat a build or pip install as including the path dependencies source files like if they had been specified via packages or include in this project’s pyproject.toml and it’s dependencies like if they had been included as direct dependencies in this project’s pyproject.toml.

In other words, for the example given in this issue doing cd ./b && poetry build would treat this as if it was a project structured like:

[tool.poetry]
name = "b"
packages = [{include = "a", from = "../a"}]

[tool.poetry.dependencies]
python = "^3.9"
# all of a's dependencies

poetry-core could do the same. Then both poetry build and pip install would work (I think, I hacked up some stuff and it builds wheels correctly for me).

The second case is a bit more complex. There’s questions about what version to use and such. Not unsolvable issues, but requiring discussion. And I think this is the minority of the use cases (also not this issue) so worth discussing elsewhere.

All of this would only work if the path dependency is a poetry project, but that seems like a reasonable limitation. It could be made opt-in/backwards compatible with something like a = { path = "../a", build-as = "{path (default), include, require" }.

adriangb on Oct 11, 2022

I think the bit of confusion here is the author of python-poetry/poetry-core#273 did a demo for project maintainers last week, and it turned out to basically be what I would consider to be a partial attempt at python-poetry/poetry#4583 – the workspace bit is really a red herring as it’s only a marker file that prevents further path traversal.

Critically, that PR does not (look at the code, since the body is still a bit confusing since the author is from a different ecosystem) canonicalize path dependencies (dependencies, not package include entries) like the ask in this issue. I think we should migrate all discussion to python-poetry/poetry#4583 of anything that isn’t the requested normalization (absolute path) of path dependencies when exported in PEP 508 format for a build.

neersighted on Oct 11, 2022