pipenv: Pipenv lock fails for packages with extras as of 2022.8.13

Issue description

There was a regression in the ability to lock Pipfiles containing packages with extras on the 2022.8.13 release. This was working fine on versions prior to 2022.8.13. I believe this is as a result of the changes introduced in #5234

Expected result

Given the Pipfile in the replicate section, it should lock, showing

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (3e8e27)!

Actual result

[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 824, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 772, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1126, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 919, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 723, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Constraints cannot have extras

Steps to replicate

With the following Pipfile

[packages]
"uvicorn[standard]" = "~=0.18.2"

[dev-packages]
pytest = "~=7.1.1"

[requires]
python_version = "3.9"

Run pipenv lock

Note that there is a package with ‘extras’, namely uvicorn[standard]. This issue does not occur with normal packages.

The change to lock the packages first and then treat that lock as constraints for the dev packages seems to raise the exception that the extras are not allowed.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 33 (22 by maintainers)

Most upvoted comments

Or maybe I’m wrong, PipDeprecationWarning says Constraints are only allowed to take the form of a package name and a version specifier so it should not containts url. In that case we could easily fix that by using new_dep.is_named.

new_dep = Requirement.from_pipfile(dep_name, dep) 
    if new_dep.is_named and is_constraints(new_dep.as_ireq()): 
        c = new_dep.as_line().strip() 
        constraints.append(c) 

But the documentation does not say that. So maybe it could have url as constraints? I tried using this constraints file with pip and it did work.

# constraints file
django-redis @ https://github.com/antoniomika/django-redis/archive/4.12.2.zip 

@dqkqd I found this interesting and has a lot of context: https://github.com/pypa/pip/issues/8210

That was a long read, but the conclusion is that “unnamed constraints are not allowed.”

I think it’s the bug in requirementlib. Actually the result from new_dep.as_line() at line 284. https://github.com/pypa/pipenv/blob/f04071c52f09015758cd3eab68fae0542c4325aa/pipenv/utils/dependencies.py#L274-L286

For example: if I passed in the packages django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"} the output I expected is django-redis @ https://github.com/antoniomika/django-redis/archive/4.12.2.zip as mentioned in https://pip.pypa.io/en/stable/reference/requirements-file-format/#example

But the actual output is https://github.com/antoniomika/django-redis/archive/4.12.2.zip, and it doesn’t contain name anymore, it’s name is None now.

The name is disappeared because: as_line call self.line_instance.get_line https://github.com/pypa/pipenv/blob/f04071c52f09015758cd3eab68fae0542c4325aa/pipenv/vendor/requirementslib/models/requirements.py#L2801-L2805

self.line_instance.get_line check if package should have a name or not: https://github.com/pypa/pipenv/blob/f04071c52f09015758cd3eab68fae0542c4325aa/pipenv/vendor/requirementslib/models/requirements.py#L219-L220

And because it has url, so it’s name is discarded from now. https://github.com/pypa/pipenv/blob/f04071c52f09015758cd3eab68fae0542c4325aa/pipenv/vendor/requirementslib/models/requirements.py#L753-L760

@dqkqd I see what you mean now, we are using that method to filter them out already, but thats causing the emitting of a deprecation warning because that method seems to expect that data was already cleaned – probably we should just have our own method for this like you were suggesting, be it a similar copy or a simplified refactor.

That begs the question of what is causing resolution failures for @antoniomika – can you elaborate on the file configuration setting you are using? Do you just mean local editable installs, or something else?

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

https://github.com/pypa/pipenv/blob/0d8662977a3b2bcb57bf551a21ce1681ad0763a6/pipenv/utils/dependencies.py#L281-L283

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

https://github.com/pypa/pipenv/blob/0d8662977a3b2bcb57bf551a21ce1681ad0763a6/pipenv/patched/pip/_internal/req/req_install.py#L854-L880

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

I don’t think we need to rewrite the pip function, I think its actually warning us of valid errors. I think the issue is that we need to filter out certain requirements from the Pipfile if they are constraints, because it ends up confusing the resolver if they get included. Probably for the extras we can do something like:

new_dep = Requirement.from_pipfile(dep_name, dep).as_ireq()
if new_dep .extras:
    new_dep .extras = None
if not new_dep.name or new_dep.editble:
    continue
....