pipenv: Missing info for package conflict

Issue description

When a dependency conflict occurs, on many occasions I see that there is not sufficient information to diagnose the conflict. This can sometimes occur when first installing a project, so I cannot run pipenv graph as no dependencies are installed (installation fails). For example, in the trace below we can see that pyspark==3.3.2 fails to install due to a conflict, but a conflict with what?

home$ pipenv install 'pyspark==3.3.2'
Loading .env environment variables...
Installing pyspark==3.3.2...
Resolving pyspark==3.3.2...
Adding pyspark to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock (3ffa3e) out of date, updating to (2149eb)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
✘ Locking Failed!
⠏ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Cannot install -r /var/folders/ts/ypl0gvnn55z1y7gydrghpdxc0000gn/T/pipenv-2dq4knuw-requirements/pipenv-sdc6xfpf-constraints.txt (line 10), -r /var/folders/ts/ypl0gvnn55z1y7gydrghpdxc0000gn/T/pipenv-2dq4knuw-requirements/pipenv-sdc6xfpf-constraints.txt (line 12) and pyspark==3.3.2 because these package versions have conflicting dependencies.
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 704, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 662, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:       ^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 642, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 1167, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 948, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.7.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 690, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Expected result

The logs will show why the conflict occurs, i.e. what packages are conflicting.

Details

pipenv, version 2023.7.23

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 1
  • Comments: 15 (10 by maintainers)

Most upvoted comments

Analysis for Issue #5823:

Analysis of Pipenv Issue #5823: Missing info for package conflict

1. Main Problem

The issue reports that Pipenv sometimes fails to provide enough information about dependency conflicts during package resolution, especially when installing a new project. This makes it difficult for users to diagnose and resolve the conflict, as they are not told which packages are causing the issue.

2. Comments Discussion

  • User sdepablos provides a concrete example with a Pipfile that consistently produces this error.
  • Maintainer matteius suggests using the --verbose flag to get more information.
  • User boonware confirms that even with --verbose, the conflict information is still missing.
  • User kalebmckale requests that the temporary files (which contain the conflicting package information) should be either kept or logged.
  • Maintainer matteius explains that the problem is due to the resolver running in a subprocess, which causes its output not to be properly relayed back to the main process. They also mention a draft branch that aims to solve this by running the resolver in the parent process.
  • Maintainer matteius claims the issue is solved in a new release. However, user shayneoneill and agalazis disagree and request reopening the issue.
  • User agalazis clarifies that using --verbose does reveal the conflict information, but suggests improving the error message to guide users towards using --verbose.

3. Proposed Resolution

The root cause seems to be the lack of proper error handling and communication between the main Pipenv process and the subprocess running the resolver. This can be resolved by:

  1. Improving error reporting: Catch the ResolutionImpossible exception raised by the resolver subprocess and parse its “causes” attribute to extract the conflicting packages.
  2. Displaying conflict information in the error message: Include the names of the conflicting packages in the error message presented to the user.
  3. Guiding users towards using --verbose: If the conflict information is not available (e.g., due to a bug in the subprocess communication), the error message should explicitly advise users to try again with --verbose for more details.

4. Potential Code Changes

This snippet shows a potential way to implement the proposed changes, focusing on error handling:

# pipenv/utils/resolver.py - inside venv_resolve_deps() function
try:
results, hashes, internal_resolver = actually_resolve_deps(...)
except ResolutionFailure as e:
error_message = str(e)
# Try to extract conflict information from the exception.
try:
causes = e.__cause__.causes
conflicting_packages = [
cause.requirement.name for cause in causes
]
error_message += (
f"\nThe following packages have conflicting dependencies: "
f"{', '.join(conflicting_packages)}"
)
except AttributeError:
# Conflict information not available, advise using --verbose.
error_message += (
"\nPlease try again with the '--verbose' flag for more details."
)
raise ResolutionFailure(message=error_message)

5. Additional Steps

  • Thorough testing: The proposed changes need to be tested extensively with different scenarios and Pipfiles to ensure they correctly identify and display conflict information.
  • Consider PIPENV_RESOLVER_PARENT_PYTHON: Investigate if merging the draft branch mentioned by matteius (running the resolver in the parent process) would simplify error handling and provide more reliable conflict information.
  • Documentation: Update Pipenv documentation to clarify the use of --verbose and explain how to interpret the conflict information displayed in the error message.

By implementing these changes, Pipenv can become more user-friendly by providing clearer and more actionable information about dependency conflicts.

Whats in the new release that resolves it?. Still does this for me and I’m mystified as to what actually is the thing thats conflicting. Like, couldnt it just say “Hey this thing wants version x but that thing wants version y”?

I pushed a change to the future looking pipenv branch which get the -v flag working with the pip resolver again in a more expected way: https://github.com/pypa/pipenv/pull/5793

@boonware 2023.7.11 had an unintentional side effect where the resolver wasn’t being run in the target python version. You can enable this “feature” in 2023.07.23 by setting env var: PIPENV_RESOLVER_PARENT_PYTHON = 1

I think the problem is the error logging in the sub-process doesn’t make it to the parent process.

That being said, it maybe the same, but could you also check the behavior with: https://github.com/pypa/pipenv/pull/5793