pipenv: Missing info for package conflict
Issue description
When a dependency conflict occurs, on many occasions I see that there is not sufficient information to diagnose the conflict. This can sometimes occur when first installing a project, so I cannot run pipenv graph as no dependencies are installed (installation fails). For example, in the trace below we can see that pyspark==3.3.2 fails to install due to a conflict, but a conflict with what?
home$ pipenv install 'pyspark==3.3.2'
Loading .env environment variables...
Installing pyspark==3.3.2...
Resolving pyspark==3.3.2...
Adding pyspark to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock (3ffa3e) out of date, updating to (2149eb)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
✘ Locking Failed!
⠏ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Cannot install -r /var/folders/ts/ypl0gvnn55z1y7gydrghpdxc0000gn/T/pipenv-2dq4knuw-requirements/pipenv-sdc6xfpf-constraints.txt (line 10), -r /var/folders/ts/ypl0gvnn55z1y7gydrghpdxc0000gn/T/pipenv-2dq4knuw-requirements/pipenv-sdc6xfpf-constraints.txt (line 12) and pyspark==3.3.2 because these package versions have conflicting dependencies.
[ResolutionFailure]: File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 704, in _main
[ResolutionFailure]: resolve_packages(
[ResolutionFailure]: File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 662, in resolve_packages
[ResolutionFailure]: results, resolver = resolve(
[ResolutionFailure]: ^^^^^^^^
[ResolutionFailure]: File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 642, in resolve
[ResolutionFailure]: return resolve_deps(
[ResolutionFailure]: ^^^^^^^^^^^^^
[ResolutionFailure]: File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 1167, in resolve_deps
[ResolutionFailure]: results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]: ^^^^^^^^^^^^^^^^^^^^^^
[ResolutionFailure]: File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 948, in actually_resolve_deps
[ResolutionFailure]: resolver.resolve()
[ResolutionFailure]: File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 690, in resolve
[ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Expected result
The logs will show why the conflict occurs, i.e. what packages are conflicting.
Details
pipenv, version 2023.7.23
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 15 (10 by maintainers)
Analysis for Issue #5823:
Analysis of Pipenv Issue #5823: Missing info for package conflict
1. Main Problem
The issue reports that Pipenv sometimes fails to provide enough information about dependency conflicts during package resolution, especially when installing a new project. This makes it difficult for users to diagnose and resolve the conflict, as they are not told which packages are causing the issue.
2. Comments Discussion
sdepablosprovides a concrete example with a Pipfile that consistently produces this error.matteiussuggests using the--verboseflag to get more information.boonwareconfirms that even with--verbose, the conflict information is still missing.kalebmckalerequests that the temporary files (which contain the conflicting package information) should be either kept or logged.matteiusexplains that the problem is due to the resolver running in a subprocess, which causes its output not to be properly relayed back to the main process. They also mention a draft branch that aims to solve this by running the resolver in the parent process.matteiusclaims the issue is solved in a new release. However, usershayneoneillandagalazisdisagree and request reopening the issue.agalazisclarifies that using--verbosedoes reveal the conflict information, but suggests improving the error message to guide users towards using--verbose.3. Proposed Resolution
The root cause seems to be the lack of proper error handling and communication between the main Pipenv process and the subprocess running the resolver. This can be resolved by:
ResolutionImpossibleexception raised by the resolver subprocess and parse its “causes” attribute to extract the conflicting packages.--verbose: If the conflict information is not available (e.g., due to a bug in the subprocess communication), the error message should explicitly advise users to try again with--verbosefor more details.4. Potential Code Changes
This snippet shows a potential way to implement the proposed changes, focusing on error handling:
5. Additional Steps
PIPENV_RESOLVER_PARENT_PYTHON: Investigate if merging the draft branch mentioned bymatteius(running the resolver in the parent process) would simplify error handling and provide more reliable conflict information.--verboseand explain how to interpret the conflict information displayed in the error message.By implementing these changes, Pipenv can become more user-friendly by providing clearer and more actionable information about dependency conflicts.
Whats in the new release that resolves it?. Still does this for me and I’m mystified as to what actually is the thing thats conflicting. Like, couldnt it just say “Hey this thing wants version x but that thing wants version y”?
I pushed a change to the future looking pipenv branch which get the
-vflag working with the pip resolver again in a more expected way: https://github.com/pypa/pipenv/pull/5793@boonware
2023.7.11had an unintentional side effect where the resolver wasn’t being run in the target python version. You can enable this “feature” in2023.07.23by setting env var:PIPENV_RESOLVER_PARENT_PYTHON = 1I think the problem is the error logging in the sub-process doesn’t make it to the parent process.
That being said, it maybe the same, but could you also check the behavior with: https://github.com/pypa/pipenv/pull/5793