pip: `pip install .` leaves a build directory that is causing issues for some tools

It’s worth noting that this affects more than pytest – git, docker, linters, IDEs, other test runners, etc. Of course they all have options to ignore a directory, but it’s a hassle. It would be better if there was an option (or default) for pip to clean up the build directory after installing.

I’d suggest filing an issue w/ pytest, to ask for a change to the discovery logic – ignore the build directory by default.

I will say, I think that most or all of the breakages we ran into came up during the PEP 517 as things that were likely to be possible sources of breakages (for example). Maybe we need to do a better job of summarizing these discussions or when making changes related to past discussions going back and reviewing past discussions or something.

I’ve long believed that beta releases for tools like pip are of marginal benefit. When we’ve tried to do them in the past we’ve rarely caught many if any issues, even show stopping ones, until a final release was cut. That’s why I stopped doing them when I was still releasing pip and instead tried to be available to quickly react to broken releases. Attempting to phase large scale changes in over time is also a good pattern (with many ways to handle it like dark reads, opt in or opt out flags, etc).

I don’t think “blame” is the most useful thing to focus on. One could easily argue that everything was working fine until pip broke backwards compatibility by no longer ensuring builds were done in an ephemeral location. Unfortunately Python’s packaging has hit Hyrum’s Law to a pretty large degree, so it’s super common that anytime we make any changes to long standing behavior we’re going to be breaking compatibility with something. The only real solution to it is to be more deliberate and careful with rollback plans and provide ways to work round etc the more risky a change is, but to fundamentally accept that these things are going to happen as well.

I’m really curious how folks here, think we should communicate around potentially disruptive stuff like this. We had a beta period for getting feedback on this change, that was publicized an order of magnitude more than we’ve ever done in the past, and we still got no reports from users that would even hint at how disruptive this could be to their workflows.

We’ve been bitten by these things in the past (like back in 2018, with pip 10.0) and if you have any thoughts or feedback on how we can do communication around changes, and get feedback on those changes, please chime in at #7628. Everything from “here’s where I go to look for information, to I don’t expect pip to ever change, to I don’t look for information ever” is useful input to us.

Presumably the workaround is simply to use the --ignore=build flag to pytest?

Thanks for all the reports folks! For anyone who hasn’t been following this closely, a bunch of related discussion has taken place at https://github.com/pypa/pip/issues/7555#issuecomment-625362521.

If you’re wondering why this change was made: pip’s copy-to-a-temporary-directory-and-build-there logic has been a source of a lot of bugs (performance issues, correctness issues and more) over the years. We made this change to improve the user experience in this area, to solve those issues by removing that band-aid solution for something much simpler and less fragile. With hindsight, I think we pulled the band-aid out a bit too early/quickly. Although we expected there to be some workflows that’d break, we’d expected the breakages would occur in more niche usecases, and not be as broadly affecting as they have been.

In summary, this change is being lot more disruptive than we had initially anticipated and we’ll be reverting the “build in place” behavior change in a pip 20.1.1 release. We’ll be pursuing a different avenues to push build tools to stop making the assumption that we’ve broken with this change (that they can put whatever crap I want in the package’s root directory), likely by collaborating on mechanisms like the one proposed in https://discuss.python.org/t/2303/.

That’s a hard one. For some changes you can add a deprecation warning for a while when user is doing something that are not (longer) supposed to be doing. The problem with this one is that there is no way to gradually warn about it.

One possibility is something along the lines of asking people to explicitly opt into the new flow by introducing a command line flag / config file option. This can be accompanied by an active hint to the user when they run without making a choice.

The warning might look like:

pip is moving toward inplace build method which may be disruptive in some cases.
We are collecting feedback to improve the feature. Please opt in to the new flow using the inplace=true config option or --inplace=true command line flag and report any issues.

We are planning on making this the default in a future version, and eventually deprecate and remove the current approach (--inplace=false).

To turn off this warning, make an explicit selection in your command line or config file.
If you disable this, please let us know via <preferred communication channel>.

I found another problem caused by this issue: it breaks manylinux builds which are based on the official manylinux demo project. It creates wheels which can’t be installed because of ABI incompatibility!

The demo iterates over many different pythons and runs pip wheel . repeatedly. The problem arises when the following happens:

1. it runs /opt/python/cp27-cp27m/bin/pip wheel .: this creates a build directory in the cwd

2. it runs /opt/python/cp27-cp27mu/bin/pip wheel .: now it finds the build dir of the previous step but somehow it fails to recognize that it was created by an ABI-incompatible version of python (m vs mu): the result is that it creates a binary wheel which can’t be installed on the python that you just use to compile it!

For a minimal example, look at this gist: it runs pip wheel twice, once with cp27-cp27m and one with cp27-cp27mu, then it tries to install/import the extension it just built. You can run it by doing this:

$ git clone https://gist.github.com/antocuni/a1404859da8a01609c90b1bb76ad5d95
$ cd a1404859da8a01609c90b1bb76ad5d95/
$ docker run --rm -v `pwd`:/foo quay.io/pypa/manylinux1_x86_64 bash /foo/build-wheels.sh

cp27-cp27m
Python 2.7.18
pip 20.1 from /opt/_internal/cpython-2.7.18-ucs2/lib/python2.7/site-packages/pip (python 2.7)
setuptools 44.1.0

creating wheel
[...]
Processing /foo
Building wheels for collected packages: foo
  Building wheel for foo (setup.py): started
  Building wheel for foo (setup.py): finished with status 'done'
  Created wheel for foo: filename=foo-0.0.0-cp27-cp27m-linux_x86_64.whl size=7748 sha256=3ec784c6363005c118d14270f24f961b3d772e10127e13772c94d9feaf9d2a99
  Stored in directory: /tmp/pip-ephem-wheel-cache-etjxV2/wheels/d5/3f/b9/c4e4497118b8a29cdc143f0da42ab610e9ddbd1025b3977721
Successfully built foo

inspecting /wheelhouse/foo-*-cp27-cp27m-*.whl
/tmp/tmp.fJKcRbpE29 /
0000000000000000      D  *UND*	0000000000000000              PyUnicodeUCS2_FromString
[...]
Processing /wheelhouse/foo-0.0.0-cp27-cp27m-linux_x86_64.whl
Installing collected packages: foo
Successfully installed foo-0.0.0
/


cp27-cp27mu
Python 2.7.18
pip 20.1 from /opt/_internal/cpython-2.7.18-ucs4/lib/python2.7/site-packages/pip (python 2.7)
setuptools 44.1.0

creating wheel
[...]
Processing /foo
Building wheels for collected packages: foo
  Building wheel for foo (setup.py): started
  Building wheel for foo (setup.py): finished with status 'done'
  Created wheel for foo: filename=foo-0.0.0-cp27-cp27mu-linux_x86_64.whl size=7751 sha256=3a093be33cb6e4435a60cfd00f9df705f3fe3398a721b767371f40d8d3fef20f
  Stored in directory: /tmp/pip-ephem-wheel-cache-9RXS7a/wheels/d5/3f/b9/c4e4497118b8a29cdc143f0da42ab610e9ddbd1025b3977721
Successfully built foo

inspecting /wheelhouse/foo-*-cp27-cp27mu-*.whl
/tmp/tmp.PLvGnWGz55 /
0000000000000000      D  *UND*	0000000000000000              PyUnicodeUCS2_FromString
[...]
Processing /wheelhouse/foo-0.0.0-cp27-cp27mu-linux_x86_64.whl
Installing collected packages: foo
Successfully installed foo-0.0.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: ./foo.so: undefined symbol: PyUnicodeUCS2_FromString

This example (and all the travis-based projects which are based on manylinux-demo) still works with quay.io/pypa/manylinux1_x86_64:2020-04-25-37c204c which is the last to include pip 20.0.2 but it’s broken by all subsequent docker images which include pip 20.1.

If you manually downgrade to pip==20.0.2 or remove the build directory between the builds, then the example works again.

I expect that many projects will start to automatically upload broken wheels as soon as they trigger a new travis-based version release, so it’s probably better to fix it soon.

Hmm, so I suppose the solution in Dockerfiles is pip install . && rm -r build

Is there an option to pip to use the temporary location as in 20.0.2?

So, still curious… what is the point of -b? It seems like it does not alter any behavior of the permanent ‘build/‘ artifact. Could it be repurposed to specify the name of that dir?

What is the expected behavior of the --build flag in this situation?

@hsharrison, that should work for you, assuming you’re using setuptools as the build backend. However, it’s worth noting that different build backends may generate different junks in different places and I think pip will wait until there’s a better standard specifying the expected behavior.

@omry Your item 1. in the proposal is exactly what the thread @McSinyx mentioned is about.

@omry, I’ve just found out about this and hope you like it too: https://discuss.python.org/t/proposal-adding-a-persistent-cache-directory-to-pep-517-hooks/2303

It seems to me that version 20.0.2 did not create build directories when doing an build-in-place

In-place builds are new in 20.1, the reason 20.0.2 did not create a build directory is because the build was done in a temporary location.