pip-tools: pip-compile fails with current master branch of pip
If you install any version of pip after cad71a711787ed66 and try to run pip-compile on any file, you will get an error in the constructor for PackageFinder. In the linked commit, it complains about unexpected keyword argument trusted_hosts and in later commits it complains about various other keyword arguments. The current master (4cf08e85c09ad213) complains about find_links (see full traceback below), failing on this line.
The latest behavior (failing with find_links) starts with in 7d08bb37a55ff7cf.
I will note that considering the fact that pip-tools is apparently making use of pip’s internal (private) API, it would be a good idea to add a CI job to build against the master branch of pip, so you can more easily catch these issues before release.
Steps to reproduce:
- Create temporary directory (optional):
mkdir -p /tmp/pip_compile_repro
cd /tmp/pip_compile_repro
- Install the latest pip from github:
git clone https://github.com/pypa/pip.git
cd pip
# In case this behavior changes in a more recent version of pip
git checkout 4cf08e85c09ad2136ff7212719bc8d126490a3cd
cd ..
pip install ./pip
- Create a
requirements.infile and attempt to compile it
echo "six" > requirements.in
pip-compile requirements.in
Actual result
Gives a traceback:
$ pip-compile requirements.in
Traceback (most recent call last):
File ".../bin/pip-compile", line 10, in <module>
sys.exit(cli())
File ".../lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File ".../lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File ".../lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ".../lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File ".../lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File ".../lib/python3.7/site-packages/piptools/scripts/compile.py", line 256, in cli
repository = PyPIRepository(pip_options, session, build_isolation)
File ".../lib/python3.7/site-packages/piptools/repositories/pypi.py", line 95, in __init__
self.finder = PackageFinder(**finder_kwargs)
TypeError: __init__() got an unexpected keyword argument 'find_links'
TL;DR:
File ".../lib/python3.7/site-packages/piptools/repositories/pypi.py",
line 95, in __init__
self.finder = PackageFinder(**finder_kwargs)
TypeError: __init__() got an unexpected keyword argument 'find_links'
Expected Result
$ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements.in
#
six==1.12.0
CC @cjerdonek in case he wants to help out.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 16
- Comments: 24 (15 by maintainers)
For users finding this issue, a workaround is to downgrade pip:
pip install 'pip<19.2'That’s just the nature of the beast. 😦
My best suggestion to reduce the problem here is to trim the space, like @cjerdonek suggested – only supporting a few “recent” versions of pip (something like last 2 minor version releases), instead of everything till like, pip 9 and all.
In thinking about this, it seems like the main important thing is that for each pip release, there is at least one version of pip-tools that users can use. Then, if someone wants to use a newer version of pip-tools, they would just possibly need to use a newer version of pip. (If someone can’t be bothered to upgrade pip, then how important is it that they be able to upgrade pip-tools?)
With this strategy, a given pip-tools version would only need to support one minor version at a time (e.g. only pip 19.3.x). And the pip-tools release cadence would basically be the same as pip’s (maybe needing to release whenever pip comes out with a new minor version, which it would have to do anyways), but releases could be done more often, too.
I confirmed that the fix works on macOS python 3.7.4 with pip 19.2.1 I think you can close i guess
Hey, guys! Thank you @atugushev for the fix. I see you made a new release on Github yesterday but it’s not in pypi yet. Do you have an ETA for this? (We’ll also downpin it for the moment)
@cjerdonek
Spending couple days to fix pip>=19.2 compatibility and refactoring the
pip-toolsafterwards, i’ve realised that it’s important to keep CI green agains pip master. Yes, as you’ve said before, it prevents from falling too far behind. But it also helps to find bugs on pip side before the release (like this https://github.com/pypa/pip/issues/6772). The only downside is we have to refactorpip-toolsside by side withpip, even in the middle of the process.Couldn’t you still update pip-tools so that CI against pip master passes, as changes to pip occur before any pip release? That would prevent pip-tools from falling too far behind and make it so that a pip-tools release would be ready to go (provided there weren’t any last minute changes to pip that break pip-tools before a pip release).
I’ll also add that I’m hoping that at some point, we’d be able to pull out the logic for PackageFinder out of pip, into a new package.
That’s obviously after the ongoing refactoring, and it’ll help to know how PackageFinder is used here since I expect pip-tools to be a consumer of that library eventually.
It would help for me to know which parts of PackageFinder pip-tools needs / uses while refactoring of PackageFinder continues.