pip: pip 19.0 fails to install packages that import to-be-installed package from CWD

[…] could someone please check if --no-use-pep517 fixes this for them?

PyInstaller installs fine with --no-use-pep517.

If it helps another example of this (via apache-airflow is pip install pendulum==1.4.4 fails, but pip install --no-use-pep517 pendulum==1.4.4 works.

The stack trace we get is similar:

Collecting pendulum==1.4.4
  Using cached https://files.pythonhosted.org/packages/85/a5/9fc15751f9725923b170ad37d6c61031fc9e941bafd5288ca6ee51233284/pendulum-1.4.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  Complete output from command /Users/ash/.virtualenvs/clean-airflow/bin/python3.7 /Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/tmprosed3kj:
  Traceback (most recent call last):
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/pip-build-env-g__m0jh6/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
      return _get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/pip-build-env-g__m0jh6/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
      _run_setup()
    File "/private/var/folders/lr/9jc9vkgn025fn6jmwm4mv4_w0000gn/T/pip-build-env-g__m0jh6/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 85, in _run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 47, in <module>
      from build import *
    File "/Users/ash/.virtualenvs/clean-airflow/lib/python3.7/site-packages/pip/_vendor/pep517/build.py", line 7, in <module>
      from pip._vendor import pytoml
  ModuleNotFoundError: No module named 'pip'

pip 19.0.2 has been released with a fix for this.

In case anyone’s wondering about the timeline, I expect we’ll be able to have a fix ready by the end of this week or early next week and make a subsequent bug-fix release of pip soon after that.

My proposal was actually that the opt-in for PEP 517 is specifying build-system.build-backend not the existence of build-system at all, and that between now and the 19.1 release, setuptools would add build_meta_legacy and pip would use it as the default backend.

I agree that in 19.1, probably if pip can’t find setuptools.build_meta_legacy, it should fall back to the old code path. That will give us minimal breaking changes while opting in the maximum number of people.

Thanks for trawling the archives for us @cjerdonek. Stating my current understanding of the problem:

1. a lot of real world setup.py files currently assume the current directory is on sys.path when they run, creating weird bootstrapping issues as projects treat themselves as an install-time dependency
2. PEP 517 explicitly says that build front ends should not implicitly add the current directory to sys.path (it doesn’t say anything about backends)
3. pip 19.0 considers pyproject.toml without a build-system section to be equivalent to a build-system section that specifies the setuptools backend
4. the setuptools PEP 517 backend doesn’t currently add the current directory to sys.path either (since getting away from point 1 above is considered a desirable future goal)
5. Two interim workarounds are available for existing projects while the default behaviours are improved: pinning pip to less than 19.0, and explicitly setting the --no-use-pep517 option. However, folks only discover those after first discovering that upgrading pip breaks their builds.

From a functional perspective, I think the key change we want to introduce is that in the “pyproject.toml without a build-system section” case, then the directory containing setup.py should end up on sys.path again. There are two main ways to handle that:

1. Get pip to do it. This has the unfortunate side effect of making the behaviour front-end dependent, and thus potentially see existing projects fail to install unless all frontends implement the same workaround
2. Get the setuptools PEP 517 backend to do it, by inspecting pyproject.toml for a build-system section, and injecting the setup.py directory into sys.path if both the section and path entry are missing. A new pip is still needed in that situation, since it has to specify a minimum version of setuptools that correctly handles the “no build-system section” case.

In the interests of getting things working for end users as quickly as possible, without painting ourselves into any unfortunate design corners, I’d actually propose a 3 step resolution:

1. Do a new pip release (19.0.1?) now that adds the extra sys.path entry for the “no build-system entry” case. At this point, the compatibility issue should largely go away for end users.
2. Do a new setuptools release that handles the sys.path addition if the front end hasn’t already done so.
3. In a later pip release, change the “no build-system entry” case to drop the special casing, and instead set a stricter minimum version for setuptools.

This proposal is based on the fact that I think the setuptools PEP 517 backend is the “right” place to handle setup.py backwards compatibility issues, but I also think that tweaking pip directly is likely to be a much simpler change in the near term, and it’s a change that will contain the problem while the more architecturally preferable fix is worked on.

I think it’s fairly common for people to import something from the current directory into a setup.py, and just generally treat things as if setup.py is in $PWD.

I think it’s reasonable to push this responsibility onto setuptools, since that’s probably the only project that really needs it.

@jce94, use pip<19 for now.

Even if we make pep517 opt-in, this is a behavior change that was not announced and will therefore break libraries even if they already opted in (see PyInstaller, for example). In the case where a library declares an opt-in to pep517 builds and imports things locally that it expects to find on sys.path, there is no assumption being made by pip (because the library is explicit) but the library is still suddenly broken.

In that case I really don’t see an alternative besides just including cwd in setuptools, because this is just broken. Unless the proposal is to tell people to go back and fix the releases they’ve cut in between pep517 and pip 19, which, if any user has those versions strictly pinned, may suddenly stop being installable, I really feel we should consider the impact of these decisions on the user experience. Based on this discussion and the current proposals some of these libraries will not be installable with new versions of pip + setuptools going forward using the defaults unless pep517 builds are explicitly disabled.

This is pretty impactful if you’re just trying to install a package with the tooling that is provided to you by python, but actually it can’t install things somewhat randomly. I say this to draw the focus off of the technical aspects for a moment and onto the impact to the end user who may get frustrated with the tooling, the ecosystem, the libraries, or the language itself, because suddenly (and yes, only under specific circumstances), things they could install just fine now can’t be installed. I really do think we should close this gap in any solution that is implemented.

We currently use a failure stack to handle failed installations in pipenv and I’m adding --no-use-pep517 when available to handle failures as a result of these changes. I’m not sure that will be intuitive to the average user, since it’s probably not even immediately clear what the cause of the problem is. I say this just to point out that we have a workaround, but it feels important to try and close this gap to help users out a bit on this one

(edit: also big thanks to pganssle, cjerdonek, pfmoore, pradyunsg, ncoghlan, and everyone else who has been putting in a bunch of time and effort on this)

With both approaches operational on my local machine, I tested both #6210 and #6212 against the original problematic PyInstaller==3.4 requirement.

• #6210 worked as desired: https://github.com/pypa/pip/pull/6210#issuecomment-458144828
• #6212 is not behaving as expected: https://github.com/pypa/pip/pull/6212#issuecomment-458148519

I don’t know whether the #6212 failure is a test setup problem, or if there’s actually a further problem in the setuptools pre-release, I only know that there’s further integration work to be done before we can be confident in that solution, whereas I’m confident in #6210 now - the only thing wrong with it is that it’s a horrible compatibility hack that we don’t want pip to have to carry forever.

I would replace setup.py with package.json and just add Python section for it instead of yet another package description format.

At least then I can use ==1.x version specifiers.

Just as a note, I don’t think PEP 517 ever had as a design goal that all build tool access would be possible via the hooks.

This is not what I suggested, and it’s a bit of a digression. My point was that the problems that necessitated PEP 518 also exist more generally for all setup.py commands, and those are best solved by moving people away from invoking setup.py at all. No standard is necessary in order to do this in the same way flit does not need a PEP to add subcommands.

So I skimmed a bunch of the emails, and I think you can at least skip ahead to August 29, where Nick seems to be suggesting there is consensus on leaving the source directory out of sys.path: https://mail.python.org/pipermail/distutils-sig/2017-August/031413.html (One by one, people were becoming convinced of Nathaniel’s arguments.)

However, in this same email linked above, Nick does say the following 😃

1. If omitting it is genuinely a problem, we’ll likely find out soon enough as part of implementing a setup.py backend

Here is a fuller paragraph from the email:

So I think we can deem this one resolved in favour of “Frontends must ensure the current directory is NOT on sys.path before importing the designated backend”, as starting there will mean we maximise our chances of learning something new as part of the initial rollout of the provisionally accepted API.

I don’t know yet if there are later emails that alter this summary, but there aren’t a whole lot of emails on the topic after.

I’ve published a shim that implements the sys.path fix.

That’s awesome! Regardless of the ultimate fix, this is a really nice example of the flexibility of the PEP 517 hook system 😃

For package maintainers looking to resolve this issue for your users: I’ve published a shim that implements the sys.path fix.

https://pypi.org/project/setuptools-localimport/

Hopefully this can work as a stopgap so we can ponder how this should be moved forward without rushing into a solution, or unnecessarily slow down the adoption of pip 19.0 (which contains much more goodies than just PEP 517).

I don’t think this is a bug in pip/setuptools, from my reading of PEP 517’s Build Environment section it seems nothing about an environment should be assumed except that the dependencies declared in pyproject.toml are available.

Also isn’t importing the package being installed from setup.py is a bad practice? There are much better ways of maintaining package version in one place, e.g. as described in this packaging guide from PyPA.

As @cjerdonek mentioned in https://github.com/pypa/pip/issues/6175#issuecomment-456769285, could someone please check if --no-use-pep517 fixes this for them?

I suspect the cause of this issue is that build isolation or the PEP 517 code isn’t making sure that the root of the package directory is on the sys.path, because pandas has a versioneer.py sitting next to setup.py. I recall this coming up at some point, but I don’t remember off the top of my head what that discussion was. This might be considered an issue with the setuptools build backend instead of pip, or it might be the fault of pip’s isolation mechanism.

In a few hours. 😃

See the pinned issue.

The new version of setuptools, version 40.8.0 is now available with the build_meta:__legacy__ backend.

I was unable to resolve this issue with the suggested workarounds while working with pipenv. Freezing pip to 18.1 in the Pipfile seems to have no effect as pipenv keep forcing latest pip version. I can manually set pip to 18.1 but when I recreate the pipenv virtual environment Pipenv would upgrade to the latest pip no matter what…Any recommendations to make it stick?

New PR #6229 that:

1. Implements @pganssle’s suggested interim workaround of making a build-system section in pyproject.toml the initial requirement for automatically opting in to PEP 517 (deferring the “any pyproject.toml file” opt-in to a later release)
2. Adds 3 dedicated test cases covering importing an adjacent package from setup.py

I’m going to close the other 2 PRs in favour of that one, since it’s the minimal fix that should get things working again for end users, and doesn’t require any horrible hacks like #6210 or a new setuptools release like #6212

Working on #6210 has let me answer my own question from above about how hard it will be to address this at the pip level: the challenge is that the information about whether or not the source directory should be inserted as sys.path[0] needs to be tunnelled from the code that reads pyproject.toml through to the PEP 517 hook caller, and then from there into the actual in-process wrapper script.

Those are doable without major architectural changes (a pip._implicit. prefix on the build backend name for the first part, and a PEP517_SYS_PATH_0 env var for the latter), but it means temporarily modifying the vendored pep517 code until the proper fix in setuptools is ready.

I’ve created the build_meta_legacy backend in pypa/setuptools#1652. I would really prefer it if pip would switch to using setuptools.build_meta_legacy as the default backend, but I think creating a built-in “legacy shim” backend is about as far as I’m comfortable going in setuptools. I do not want to get stuck indefinitely supporting full “python setup.py install emulation” in the main setuptools PEP 517 backend.

How would that work with --no-binary :all:?

Personally, I agree with the current approach of relying on the existence of pyproject.toml file. The issues IMO stems from people using pyproject.toml for things other than packaging. The correct way out is therefore to push non-packaging tools to offer another way for configuration, so people can choose whether to use pyproject.toml or not.

success for me too

pip install pyinstaller --no-use-pep517
Collecting pyinstaller
  Using cached https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz
Requirement already satisfied: setuptools in c:\python37\lib\site-packages (from pyinstaller) (39.0.1)
Collecting pefile>=2017.8.1 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/ed/cc/157f20038a80b6a9988abc06c11a4959be8305a0d33b6d21a134127092d4/pefile-2018.8.8.tar.gz (62kB)
    100% |████████████████████████████████| 71kB 1.0MB/s
Collecting macholib>=1.8 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Collecting altgraph (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
Collecting pywin32-ctypes (from pyinstaller)
  Using cached https://files.pythonhosted.org/packages/9e/4b/3ab2720f1fa4b4bc924ef1932b842edf10007e4547ea8157b0b9fc78599a/pywin32_ctypes-0.2.0-py2.py3-none-any.whl
Collecting future (from pefile>=2017.8.1->pyinstaller)
  Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)
    100% |████████████████████████████████| 829kB 1.6MB/s
Installing collected packages: future, pefile, altgraph, macholib, pywin32-ctypes, pyinstaller
  Running setup.py install for future ... done
  Running setup.py install for pefile ... done
  Running setup.py install for pyinstaller ... done
Successfully installed altgraph-0.16.1 future-0.17.1 macholib-1.11 pefile-2018.8.8 pyinstaller-3.4 pywin32-ctypes-0.2.0

Ok, then that’s certainly an issue with the new PEP 517 code and I’m pretty sure the issue is just that the directory containing the project root hasn’t been added to sys.path. Maybe @pfmoore will have a better sense of if that should be pip’s responsbility or setuptools.