pip-tools: requirements resolution does not respect include-system-site-packages = true and break base installation

Environment Versions

  1. Windows 10 enterprise last update
  2. Python version: 3.9.5
  3. pip version: 21.3.1
  4. pip-tools version: 6.4.0

I’m using Winpython a scientific distribution (more than 560 packages for almost 7Gb) as a base installation. It’s an accurately crafted distribution that guarantees compatibility among all the installed packages. I’m using that precisely because i don’t want to go hunting for incompatibilities between the last versions of packages like jupyter, ipython, ipykernel, numpy, pandas, pytorch, tensorflow, etc and the tenth of packages that go around them. Another thing i don’t want is that each venv for each project ends up containing the very same packages already present in the base installation using then Gb and Gb of useless space and that’s the reason why i use include-system-site-packages = true in my venv’s.

I’ve always managed all with base commands like python -m venv and pip install but i wanted to automatize a bit the thing so i gave in sequence a try to pipenv and poetry, but i had to quit due to the same problem i found here in pip-tools.

Steps to replicate

As an example, if i try to install a very simple thing like blackcellmagic in a bare venv with pip i get installed only the 4 requirement packages missing in the base installation, because the rest 60 packages of requirements chain are already satisfied by the base installation, (remember the include-system-site-packages = true). If i try pip-compile with the same requirement i get 66 (sixty-six) packages because the whole requirements chain is always updated to the last version ignoring the fact that it really it is already satisfied, so breaking the compatibility (both at pip requisite level and at execution level) with a lot of other packages in the base installation and making the venv a useless disk space filler.

I tried both --no-build-isolation and --no-upgrade options, but evidently either i didn’t understand their meaning or I’m i missing something, and i didn’t find any other way to get pip-compile resolver to honor include-system-site-packages = true like pip does.

I’m expecting there is a way to force pip-compile to accept the already installed satisfied requirements, like pip, instead of unconditionally always forcefully updating them to the last version so breaking compatibility with base installation.

I’m not interested to pursue this with pipenv and poetry because i discovered they are both a sort of overkill for my needs, while pip-tools, if i get this problem solved, would completely satisfy my needs to streamline a lot of hand management.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

The current design of this tool does not have pip-compile manage the environment’s installed packages, instead leaving that to pip-sync or pip or whatever.

pip-compile only manages the dependency lists, which may inform each other, but are not informed by the environment’s installed packages.