pipenv: pipenv lock --keep-outdated does not respect Pipfile.lock constraints

@matteius

I noticed you thumbs downed my last comment, but its the truth.

I believe this is more of your own opinion rather than universal truth. I noticed that you and probably other pipenv maintainers have a fundamentally different understanding of how a dependency management system should work when it comes to package installation. Since there are so many differences here, it seems that the Python community needs another system as an alternative to pipenv. It’s a pity, because it’s always more effective to focus all efforts and budget on improving one solution, as was the case with bundler for Ruby. Perhaps poetry will be developed in the direction expected by the other part of the community that has spoken on this and similar issues. E.g. see https://github.com/pypa/pipenv/issues/3150#issuecomment-522947210 as this comment sums up the expectations of “the other part” of the community.

I think I’ll end my participation in this discussion here because neither I nor others present here (and in related issues) will convince you to change the design decisions regarding the installation, nor, as you have probably noticed, you will not convince others.

Also the fact that pipenv install alters other, unrelated package dependencies sounds like “feature envy”.

The other unrelated packages are actually related to your Pipfile and the specifiers you install there. All dependencies may affect the resolution of the other dependencies and so a complete resolver phase has to happen and this is not just my opinion, without this, you get the kind of behavior that --keep-outdated provides which leads to plenty of issue reports because the resolver results were not respected. The only input into the resolver is the Pipfile and its the end user’s responsibility to pick reasonable specifiers for their top level packages and things they care to pin.

even if I specify pymssql = “~=2.2” and thenpipenv install numpy it may upgrade pymssql , which I then have to test separately and what’s more, if by any chance pymssql minor upgrade introduces a “bug”, which may always happen, it will make debugging harder (it could be avoided in the first place).

it could be avoided in the first place by recognizing this fact (which you have) and then pinning it in your Pipfile until you are ready to put the time into testing the upgrade of pymssql.

A flawed implementation does not mean the whole concept is wrong. At least pipenv should have a sane option of keeping the unrelated packages untouched.

This is not how the pip resolver needs to work. you think the packages are unrelated, but they are related to the other packages you are resolving/installing and those packages apparently did not restrict it in the way you want, so please use the Pipfile for this purpose.

The common problem theme here with --keep-outdated is that “I wanted pipenv to do what I want it to, but I don’t want to tell it how.” There is no magic forumla to figure out that you want your pymssql pinned, but that you want all of your dependencies to all resolve and install correctly, so you have to be explicit and not look for a magic bullet like --keep-outdated. We are likely to deprecate this flag in the near future and remove it after that.

@matteius apologies for splitting your comment in parts, but it’ll be easier to reply to a couple of statements you made this way:

it should never touch other packages in Pipfile.lock unless they are direct dependencies of the newly installed package.

This isn’t true,

I believe this is your private opinion, but as I’m coming from ‘bundler’ world in Ruby I can confirm that this at least is true there, to back it up with something, I’ll at least mention the “principle of least surprise”, which pipenv is currently violating. Also the fact that pipenv install alters other, unrelated package dependencies sounds like “feature envy”.

This also resembles the “git commit” situation. Compare git add . or git add -A with precisely selecting files and changes. It seems to me that it is generally accepted that a commit should contain an “atomic” change in the system, and I would treat package dependencies management similarly. pip install packageA with an accidental upgrade of unrelated packageB is like adding “all” changes in working directory to a specific commit in git.

What you should be doing instead is actually picking proper specifiers for your project in the Pipfile such that you are not inconvenienced by re-locking when installing new packages

Most people do that already, but still, even if I specify pymssql = "~=2.2" and thenpipenv install numpy it may upgrade pymssql , which I then have to test separately and what’s more, if by any chance pymssql minor upgrade introduces a “bug”, which may always happen, it will make debugging harder (it could be avoided in the first place).

pipenv lock --keep-outdated but that is a flawed implementation and we should not prefer users to use that as the default

Lastly , this is a different story. A flawed implementation does not mean the whole concept is wrong. At least pipenv should have a sane option of keeping the unrelated packages untouched. Perhaps fixing this implementation will be a good compromise.