warehouse: Reject packages without a Requires-Python
A problem I’ve come across many times now is that a package will drop support for an old version and they’ll even add a python_requires directive to their setup.py, only to cut a release using an old version of setuptools that doesn’t actually understand the python_requires directive and thus silently fails. See, e.g. sphinx. This causes a bunch of problems because the erroneously uploaded packages now pollute pip installs of that package unless they are removed from PyPI or a post release is made.
Since there is no good reason to upload a package without explicitly specifying a Requires-Python directive, I think we can assume that this is an error and reject packages that don’t have a Requires-Python.
My recommendation for migrating to the “always throw an error” version:
- Start throwing a warning when
Requires-Pythonis missing, with a link to the documentation on how to addRequires-Python - Update
twineand all tools that upload packages (andsetuptools, even though it’s deprecated) to automatically transform that warning into an error overridable by--allow-no-python-requires - Add a backwards-compatibility API that allows you to upload packages without
Requires-Pythonby configuring your upload endpoint to something not the default - Upgrade the warning to an error, dropping support for
--allow-no-python-requires.
I think we could swap the order of 2 and 3 pretty easily - I’m guessing that updating the upload tools would be easier to do which is why I put them in this order, but 2 could be implemented in terms of 3.
In terms of time frame, I don’t know how aggressive you (the Warehouse team) want to be. I think the first 3 can happen as soon as an implementation is available. It’s probably a big ask to have all package maintainers switch over in a relatively short period of time, but I think a long deprecation period will be harmful given that it’s likely a large number of packages are going to start dropping support for Python 2 soon, which will probably cause a ton of headaches if people aren’t including Requires-Python in their metadata. Maybe a 6 months or so?
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 8
- Comments: 15 (12 by maintainers)
Commits related to this issue
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
- Added ability to publish to PyPI We now use twine and can publish our package to PyPI! Twine is required to properly push metadata to PyPI irrelevant of the underlying python versions Upstream: htt... — committed to cyberark/cyberark-conjur-cli by sgnn7 5 years ago
Well but if you are an advanced user and really want to get this package on an old Python version you probably can figure it out. Though if you are a new user and
pip install fooand it does not work you at a loss. So I would see over-enforcing as better than being too lax.And the all point of Python-Requires is also exactly for those old-python-users not to install a too-recent version of the package. Mostly I implemented it in Warehouse/PIP because IPython wanted to drop 2.x and couldn’t without breaking for old user. I would even expect the opposite of you, and have project to drop support ini CI, break on old python, and forget to update Python-Requires.
I have not been part of some of the discussion about the resolver so pardon me if I’m mistaken. But, unlike the other dependencies, pip cannot change the current version of Python right ? Requiring Python-Requires may dramatically reduce the search space of the resolver as the Python version is not a variable. It would even be strange to me if the resolver even knows about Python-Requires, wouldn’t you just immediately filter out any package that is incompatible with the current Python and give the resulting set to the resolver ?
It seems that the other half of this issue is “how can we make sure users update setuptools/twine more often?” - which would help for more than just this case.
For example:
pip install -U ...Regarding users already on older broken versions - less disruptive options (vs brownouts/blocking) might be:
Agreed. My point above (which I didn’t state well) was that
Requires-Pythonmetadata, while useful, can (in the most general sense) only be evaluated at the same time as the other dependency constraints on a package, after we’ve read the metadata. And making it mandatory doesn’t really change that code.It’s true that on PyPI the
Requires-Pythondata is available from the index, and so can reduce the search space - and this is great. But pip (and any other real-world Python resolver) has to handle other indexes, and even local directories containing distribution files. For those, there’s no saving andRequires-Pythonhas to be processed in the constraint evaluation phase, not the file selection phase. So we do the check in both places - one for efficiency, the other for correctness.Don’t get me wrong - I’m 100% in favour of better, more strictly enforced, and static metadata. I just don’t think having this piece will materially affect how we code the resolver in pip.