pip: Warn users about dependency conflicts when updating other packages

This is a mental note on a topic I realise needing a discussion while working on another issue.

Say we have package foo and bar with the following dependencies:

foo 1.0.0
    six<1.12

foo 2.0.0
    six>=1.12

bar 1.0.0
    six<1.12

bar 2.0.0
    six>=1.12

Given an environment with the followings installed:

foo 1.0.0
bar 1.0.0
six 1.11.0

and the user runs pip install --upgrade foo. What should we do? If we upgrade foo to 2.0.0, six needs to be upgraded as well (as an intrinsic requirement), but now it would conflict with bar. I can think of three possibile approaches:

Upgrade foo and six, and print an error/warning telling the user bar now has unsatisfied requirements.
Upgrade bar automatically to 2.0.0.
Telling the user everything is up-to-date, since the installed foo 1.0.0 is the latest version without conflicts.
Error out without modifying the environment, saying the upgrade would introduce incompatibilities.

Approach 1 is the simplest, but might be too difficult for the user to notice (especially on CI). This is probably not a good idea if we can avoid it.

Approach 2 looks like a good idea at first glance, but IMO may be confusing to the user. The dependency graph would be much less complex in more than one way in practice, and it would be difficult for the user to notice, or understand why a seemingly unrelated package got upgraded.

Approach 3 is “correct” in thoery, but is as unuseful to the user as pip’s famous “No matching distributions found for” error. There is clearly a newer version to upgrade to from the user’s perspective. Why is pip not finding it? Open GitHub and file a bug report.

Approach 4 is the most reasonable to me. In the above example, pip would emit something like six>=1.12 (required by foo) would cause incompatibility in bar (requires six<1.12). The downside is pip would need to do more work to interpret the resolution result (this does not fit into the resolution process IMO).

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 61 (55 by maintainers)

Commits related to this issue

docs: Emphasize that pip may break existing packages Related to #7744 . — committed to brainwane/pip by brainwane 4 years ago
docs: Emphasize that pip may break existing packages Related to #7744 . — committed to brainwane/pip by brainwane 4 years ago

Most upvoted comments

What is the practical solution to https://github.com/pypa/pip/issues/9482 though? If I’m running pip3 install -U foo, then really I want it stop and warn me before upgrading if the upgrade would break a dependency. Only if I pass an --ignore-breakages option or something would I want it to go ahead. This is highly undesirable behaviour at present.

alexreg on Feb 17, 2021

My apologies for giving an alternative that is wrong on all accounts. I hope thought that my confusion might help to show how the original message might be misinterpreted. Let me try a few more:

Warning: pip will upgrade foo. This may lead to foo’s dependencies being modified which may cause dependency conflicts.

Warning: pip will upgrade foo. This may lead to other packages being modified which may cause dependency conflicts.

If these don’t work hopefully my failure to find a better message does not mean that no one else will try to find one.

jtamagnan on Oct 27, 2020

https://discuss.python.org/t/does-pep-376-need-a-review/3269

pfmoore on Feb 15, 2020

Most package managers don’t have this problem because they keep a “manifest” describing the user’s original intention (e.g. package.json, Cargo.toml, Gemfile).

For Python, we have https://www.python.org/dev/peps/pep-0376/#requested that could serve a similar purpose.

xavfernandez on Feb 14, 2020

What is the practical solution to #9482 though? If I’m running pip3 install -U foo, then really I want it stop and warn me before upgrading if the upgrade would break a dependency. Only if I pass an --ignore-breakages option or something would I want it to go ahead. This is highly undesirable behaviour at present.

I see it the same way. The default behavior should be that the whole installation process aborts with an error message, telling the user that their python environment could get broken if they force the installation and maybe also give an overview of which part of the depenency graph will get broken, if so. However, I don’t know how far this is possible to implement. I just don’t think that it is smart to believe that the user knows what they are doing. That is kinda like offering someone a car key without telling them that - if they accept - the money for the car is withdrawn from their bank account which may break their financial situation. Never assume that users know the exact consequences of their actions.

elsamuray7 on Feb 22, 2021

FYI conda update has --update-dependencies and --no-update-dependencies, and you can set either in configuration as default. I’m not sure which one is the default, however.

Edit: It seems like update_dependencies = False is the default, so Conda is doing almost exactly what you want.

There is still a slight difference, Conda would update a package if that update would not break dependencies elsewhere, and you don’t want even that to happen. This is a reasonable scenario, but I believe Conda’s stance is you should spell out exact versions in environment.yml in that case instead. pip’s is similar (but with requirements files), and most people likely also expect that, so disallowing any package updates may be too drastic a change for pip to implement.

uranusjr on Nov 18, 2020

My instinct is we should only display this if pip actually upgraded a dependency.

ACTUALLY!

One of the things I just remembered is that if/when pip does install conflicting packages, it’ll print the warnings about them: https://github.com/pypa/pip/blob/a0ec4be98bcad63f9848c9c7b0cd2ed9afa00ff3/src/pip/_internal/commands/install.py#L560-L561

This “FYI: I detected conflicts in the final set of packages you’ll have when I’m done” logic in pip, is also run with the new resolver, which means we’re already printing some sort of relevant information toward helping the user understand the situation. I’m 100% OK to add the textual context to that error message. I’m imagining it’d look something like:

WARNING: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.
amazing-package 3.0 requires the-other-awesome-thing<2.0, but you'll have the-other-awesome-thing 3.0 which is incompatible.

(IDK what the best phrasing might be)

pradyunsg on Oct 28, 2020

My instinct is we should only display this if pip actually upgraded a dependency.

uranusjr on Oct 28, 2020

Do we want this message to be printed unconditionally? I feel like it’ll serve as noise beyond the “first time you see it” and acts toward conditioning users to ignore warning messages from pip.

pradyunsg on Oct 28, 2020

We discusssed this in our meeting last week and agreed to go with approach #2: “Pip only considers the packages being installed, and may break installed packages.” Or, in more words: Pip will only consider the packages being installed, and may break installed packages. It will not guarantee that your environment will be consistent all the time. If you install x and then y, it’s possible for the y you get to be different than it would be if you had run “install x y” in a single command.

@pradyunsg is also interested in thinking about directing users who might want strict mode, so they can provide “I want that” inputs on that in the beta. This could involve the CLI printout, or something in documentation. I saw there was a TODO:

Pradyun to file a tracking issue, documenting this and what we’re going to do here.

Pradyun, did you already do this?

brainwane on Jun 29, 2020

(Read pip source) “Oh that’s why. But WHY.”

uranusjr on Apr 23, 2020

That sounds good to me!

As long as we’re satisfying “it will no longer install a combination of packages that is mutually inconsistent” promise for the resolver’s behaviors, for the common cases, I’m a not-cranky kiddo.

pradyunsg on Apr 22, 2020

@ei8fdb ^

uranusjr on Apr 7, 2020

if not installed:

install whatever works “best” -> best == newest compatible version.

I believe this would “simply work”[*] in practice. Assuming the user already has a “working” environment, a package being not installed means it’s not depended by any existing distributions. The resolver can choose any version it needs to depending on the newly requested packages.

[*]: A working environment does not necessarily mean the environment does not contain any conflicts or broken dependencies, but there’s nothing wrong as far as the user concerns. pip does not need to actively fix the environment unless the user specifies so.

if already installed:

if not directly depended on by the package:

“don’t touch it”: error out when it’s incompatible with all the potential choices.

Q: do we care about telling the user “hey, pip picked an older version than it would’ve if you didn’t have X in your environment”, in cases where stuff worked out?

if directly depended upon by the package:

prefer existing installed version when possible, and allowed to change the version.

TODO: do we allow downgrades and upgrades? do we want to treat those differently?

With REQUESTED out of the consideration for the short term, I feel pip is not in the position to switch behaviour based on reverse dependency information. It is still very often wrong to automatically upgrade a package, even if it is depended by another. Django packages, for example, usually depend on django, but I’d be very annoyed if installing the latest django-debug-toolbar automatically upgrades my Django installation from 2.2 to 3.0 because djangorestframework also depends on django. The better behaviour is to never touch an already-installed package, and always error out if that does not work. It would also be extremely useful to tell the user to either upgrade or uninstall the package, and what other packages depend on the conflicting package; otherwise the user would be hard-pressed to decide what action is best.

It would be way too restrictive to always error out, of course. My feeling is this should be where --upgrade comes into play. Upgrading (or downgrading; any version-changing operations should be treated the same in this context IMO) can happen if the user supplies this flag, and --upgrade-strategy affects whether to prefer upgrading the world, or the smallest possible set.

This leaves only one question, whether we should error out, or simply warn if an incompatibility would occur (for a package in the environment, but not a dependency of the given packages in the install --upgrade call) after resolution. The checks would be exactly the same (pip already implements it at install time); the only difference would be whether we install or not. I feel the current behaviour is actually useful sometimes, but personally don’t really care either way.

uranusjr on Apr 7, 2020