pip-tools: --upgrade-packages fails to constrain subdeps, if either absent from a preexisting output file, or if --upgrade is also passed

The documentation states that the --upgrade and --upgrade-packages can be used together, however the --upgrade-packages flag is ignored when --upgrade is supplied.

Environment Versions

OS Type: macOS and Ubuntu 18.04
Python version: python 3.9.0
pip version: pip 21.3.1
pip-tools version: pip-compile 6.4.0

Steps to replicate

Using different args to the documentation, as requests is not in the django requirements, and requests<=3.0.0 does not affect the “latest” release.

# Create requirements.in file
echo "django" > requirements.in
# Run pip-compile
pip-compile --upgrade --upgrade-package 'sqlparse<=0.4.0'

Expected result

asgiref==3.4.1
    # via django
django==4.0.1
    # via -r requirements.in
sqlparse==0.4.0
    # via django

Actual result

asgiref==3.4.1
    # via django
django==4.0.1
    # via -r requirements.in
sqlparse==0.4.2 
    # via django

sqlparse is not restricted to <=0.4.0. If I omit the --upgrade flag, only supplying --upgrade-packages 'sqlparse<=0.4.0', I get the expected result (but my other packages are not upgraded to later versions). This also doesn’t work if the argument is sqlparse==0.4.0.

My current workaround is just to delete the requirements file(s) before running pip-compile, and omitting the --upgrade flag so the existing files aren’t used as a cache. I suspect either the --upgrade-packages constraint logic is inside the if not upgrade... conditional, or that --upgrade-packages only extend, not constrain the complete set of packages.

This feature should be permitted, or the documentation changed to avoid stating that this is possible

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (12 by maintainers)

Most upvoted comments

@jammie19 If I understand the question, then yes.

Here’s behavior on master, using the pip-tools repo itself as the setup.py package to test:

$ pip-compile setup.py -U 2>&1 | grep tomli
tomli==2.0.1
$ pip-compile setup.py -U -P 'tomli<2.0.1' 2>&1 | grep tomli
tomli==2.0.1

And here’s the result with #1578:

$ pip-compile setup.py -U 2>&1 | grep tomli
tomli==2.0.1
$ pip-compile setup.py -U -P 'tomli<2.0.1' 2>&1 | grep tomli
tomli==2.0.0

EDIT: I didn’t note before, but tomli is not a direct dep; it’s a dep of direct dep pep517.

AndydeCleyre on Feb 16, 2022

I’ve added some tests to validate your changes, and opened a PR into your fork. Looks like your changes solve the problem, from the test output

jamescw19 on Feb 9, 2022

I made #1578 to trial different handling of --upgrade-packages. I constructed a temporary file from upgrade_packages and parsed that in -c constraints.txt style, in addition to our existing handling of upgrade_packages.

The result is good. If it’s simple enough to do without creating a temp file, I’m interested in that.

It also presents the possibility of including the --upgrade-packages switch as a source in the annotations.

AndydeCleyre on Feb 16, 2022

I did not realize they were intended to potentially be used together, as I didn’t see sense in the sentiment “keep/install an old version of package x AND upgrade all packages to their latest” – I figured the upgrade-all bit would trump the other bit. I was wrong.

I’ll try to review this whole thing.

AndydeCleyre on Feb 4, 2022