setuptools: Please do not remove `setup.py install` as it is needed for distribution packagers

Well, it looks like we could, we would just need to create a second python package just to reimplement main() around Pep517HookCaller without calling pip first.

This is quite awkward, not least because it’s an explosion of tools and Arch Linux dislikes an explosion of custom distro tools (we devoutly avoid build recipe macros, templates, and inferred hooks with the rationale that they’re unreadable and turn the act of packaging into some secret invite-only club of people who actually understand the endless layers). It would be best if pep517 could just… not call pip when it knows all the Distributions are there. The point of this reinvention of the python packaging landscape was to make things less coupled together, right? So it seems silly to couple pep517 to pip for any reason.

Eh, things are getting even better. I’ve spent significant time trying to figure out how data_files work with wheels. Unless I’m mistaken, this is done completely outside wheel spec, and it was deprecated anyway. So if I write a useful user-facing Python program, I’m not supposed to install .desktop files for it, or a working manpage, right?

I would start with the fact that to this day there’s zero progress in including a TOML parser in Python stdlib.

I hope that I did not give you false hope that there was some possibility that we would continue maintaining the setup.py install command by suggesting that you open this issue. I am fairly confident that that will not happen for a variety of reasons. I mainly wanted to suggest that we get your requirements on the table so that we can find a better way to solve them.

Context

To give you a bit of context here, for a long time in the “distutils era”, there was not really any separation between build backends and front-ends. distutils was make + compiler + package manager + test-runner + everything else. Packages started to rely on these blurry lines between front-ends and back-ends, and we ended up with a huge morass of tightly coupled interfaces that were very hard to update for the modern world. Ideally, new build tools would come around that could solve different use cases, but unfortunately so much code out there relies on setuptools / distutils (and then layer in on top of that code that assumes that pip does the installing), that any new build tools needed to be bug-for-bug compatible with the existing stuff, in a lot of ways.

PEP 517 and PEP 518 are part of a wider effort to standardize build tooling in a way that allows for new build tools to be built without requiring every build tool to actively be compatible with every other build tool. We’ve standardized on a way to specify build dependencies (PEP 518) and standardized the relationship between back-ends (like setuptools and flit) and front-ends (like pip and tox).

setuptools’s place in the new world

setuptools has historically had features of both front-ends (e.g. installation) and back-ends (e.g. build), and since there are new de facto standards for all the front-end actions, setuptools is getting out of the business of being any sort of CLI application.

One of the biggest reasons to remove the setup.py install command is that it is broken in a large number of situations, and creates a lot of pain and confusion for end users. It creates pkg_resources-based entry points, can make your packages uninstallable, and just generally break things. It would be painful to fix and even if we did, we don’t have the resources to continue maintaining it when there’s a perfectly good and recommended alternative (pip). Just like when we removed setup.py upload and setup.py test, it may seem like we are removing something that works without replacing it, but this is wrong on both counts – there is a replacement and it doesn’t currently work, it just may not fail terribly in some use cases.

We are aware that it will require updating a lot of existing code. We’re trying our best to make it as seamless as possible, but making it seamless requires early adopters to tell us where the pain points are. Hopefully distro packagers can start trying out the alternatives and reporting where it doesn’t fit with their workflows.

The supported workflow

There is an upside to this that I hope that distro packagers can see, which is that as we move to the newly standardized interface, you can write a “build and install from source” script that will work for all projects, not just ones using setuptools. setuptools has a lot of rough edges, and I would not be surprised if you start seeing a lot more projects using flit, poetry or other build tools in the coming years.

There is no requirement that you use pip for this, though. That’s the beauty of the new standardized workflows. The workflow looks like this:

1. Check out source and build wheel from it using PEP 517 hooks.
2. Install the wheel using the process standardized in PEP 491.

There are tools to allow you to do both of these things, including the pep517 library and pip. There is also an effort underway to create a standardized installer for wheels to handle the second phase without pulling in pip: https://github.com/pradyunsg/installer

Since these are standards, you can use the tools provided that meet those standards, or you can build your own tools that meet the standards and satisfy your requirements – both choices are equally supported, at least in setuptools.

I have now a working bootstrapping script capable of building and installing the required packages to setup a Python environment in https://github.com/FFY00/python-bootstrap.

What you neglect to mention (and what caused me to waste a lot of time trying to figure out how your bootstrap works) is that you’re using a custom fork of installer. So I dare say that the tooling is not mature enough, and you had to fork it to make it work.

I’ve spent significant time trying to figure out how data_files work with wheels. Unless I’m mistaken, this is done completely outside wheel spec, and it was deprecated anyway.

I believe you are mistaken. The files that setuptools calls data_files go into a folder called e.g. foo-1.0.data/data in the wheel, and should be installed from there to whatever directory the install scheme specifies for data.A lot of the wheel spec is quite vague and relies on existing knowledge of Python packaging, but the data directory is mentioned in this section and again in this section. I’m not aware that it has been deprecated.

You can use these to install things like man pages or .desktop files, but from a developer’s point of view, it’s unreliable, because there’s no guarantee where these files end up. Most obviously, installing a .desktop file in a virtualenv doesn’t do anything (installing man pages in a virtualenv does appear to work now, so long as I activate the env). So Python developers who see Python packages as a primary distribution channel will often either avoid relying on these kind of system integrations (desktop files, systemd units, D-bus services, etc.) or provide a way to install them as a separate step after installing the Python package, rather than putting them in data_files.

And yes, I know that setup.py install is not actually removed. However, I come here by way of #2080, because I’m in a situation where setup.py install doesn’t work – people have deleted their setup.py files because setuptools’ pep517 backend supports creating wheels without them, and then we cannot do anything with those wheels, because that part of the “replace setup.py install” environment doesn’t exist yet…

This is one thing I can help you with, at least for now. I’ve written https://github.com/mgorny/pyproject2setuppy to help with packaging this stuff for Gentoo. Its ~200 lines of code, has only toml as a dep and installs everything via setuptools without requiring you to install two new huge package managers you won’t ever use or dephell which deserves the name. Of course, it will stop working when setup.py install is killed.

Besides, these PEPs pretty much prove where Python ecosystem is heading. We have new fancy way of installing stuff which uses integrated package managers in place of build systems, we have new fancy markup language that is required to deal with this… and at the same time, Python team refuses to include TOML in Python 3.9 because… the spec isn’t finished yet.

So it’s not yet ready to be included in Python so that it would reduce bootstrap trouble at least by a tiny bit. At the same time, it was mature enough to reinvent the whole build mechanics and announce killing the old one.

The easiest solution, to me, would be to always build setuptools and its dependencies using the previous version of setuptools. Presumably this or something like it is what is done for bootstrapped compilers like gcc. That said, it’s up to you how you want to solve this problem, and deferring it until it becomes an immediate threat is a reasonable choice, just wanted to give you something to think about.

We’ve discussed internally, a few times, the thought of creating a bootstrap repository for toolchains. Nothing is final.

Simply depending on the previous version of setuptools doesn’t work as that logic breaks down when moving from python 3.8 to python 3.9…

One priority for bootstrapping would be not minimizing the number of packages it takes to bootstrap. If:

• setuptools (6 packages)
• toml (after setuptools is built) plus pep517.build
• pytoml (after setuptools is built) plus flit_core
• ‘installer’, however that turns out

is a viable way forward, we’d have a couple paths to bootstrapping the ecosystem with only a handful of packages. (Graduating a battle-tested toml library to the stdlib would be super useful, truth be told. It’s become central to packaging, everything depends on one, and in fact, some things only depend on one, so it would be interesting to see what could simply build itself without any external dependencies.)

Bootstrapping pip requires 24 packages, and that rises to 40 packages if you count sphinx, which we need to build documentation in manpage format, and that’s leaving aside the issue of pip having generally undesirable behavior for packaging. We currently document various mechanisms for installing python software, with pip caveats here: https://wiki.archlinux.org/index.php/Python_package_guidelines#pip

(but using pip for this is a hassle, so no one does it, so no one focuses on documenting the pitfalls, so it might be out of date.)

Since I’m currently unsure how bootstrapping in a possible post-distutils world might look, I think we’ll have to see how the ‘installer’ package or similar tools play out before settling on one way to do it.

And in the meantime, we use setuptools, which, for our purposes at least, works essentially flawlessly.