pip: pip 19.0 fails to install packages that import to-be-installed package from CWD
Environment
- pip version: 19.0
- Python version: 3.6
- OS: MacOS
Description When running pip install pyinstaller==3.4 with pip 19.0 we are getting an install error. ModuleNotFoundError: No module named ‘PyInstaller’
Expected behavior Expect pyinstall to be installed, as it is with pip 18.1
How to Reproduce Using python3: pip install pyinstaller=3.4
Output
pip install pyinstaller==3.4
Collecting pip
Using cached https://files.pythonhosted.org/packages/60/64/73b729587b6b0d13e690a7c3acd2231ee561e8dd28a58ae1b0409a5a2b20/pip-19.0-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-19.0
(BuildVEnv) jlaroche-mbp:TrackSense$ pip install pyinstaller
Collecting pyinstaller
Using cached https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... error
Complete output from command /Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/bin/python3 /Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/tmps3z6flnv:
Traceback (most recent call last):
File "/Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
main()
File "/Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/jlaroche/Dev/uapkg/packages/system/algo/BuildVEnv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/pip-build-env-lo_ir5_f/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
return _get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/pip-build-env-lo_ir5_f/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
_run_setup()
File "/private/var/folders/j6/7t8sg1vj4q97zhh9z5cdmxbm4rz935/T/pip-build-env-lo_ir5_f/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 85, in _run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 20, in <module>
from PyInstaller import __version__ as version, HOMEPATH, PLATFORM
ModuleNotFoundError: No module named 'PyInstaller'
Maintainer note on timeline: See https://github.com/pypa/pip/issues/6163#issuecomment-460563963
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 20
- Comments: 89 (64 by maintainers)
Commits related to this issue
- Tests: CI: Temporary pin pip to < 19.0. pip 19.0 fails to install PyInstaller, see pypa/pip#6163. Until this is solve d (or another solution is provided), pin pip. — committed to htgoebel/pyinstaller by htgoebel 5 years ago
- Tests: CI: Temporary pin pip to < 19.0. pip 19.0 fails to install PyInstaller, see pypa/pip#6163. Until this is solve d (or another solution is provided), pin pip. — committed to htgoebel/pyinstaller by htgoebel 5 years ago
- Don't install pyinstaller and s3cmd in jobs that don't need it This will also probably circumvent the pip 19.0 pyinstaller [bug](https://github.com/pypa/pip/issues/6163) that hit us in all Travis job... — committed to LefterisJP/raiden by LefterisJP 5 years ago
- Try unbreaking PyInstaller installation on Windows See https://github.com/pypa/pip/issues/6163 — committed to gridsync/gridsync by crwood 5 years ago
- [AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 There is a bug or a new feature that causes a number of our dependencies to fail to install. — committed to ashb/airflow by ashb 5 years ago
- [AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 (#4576) There is a bug or a new feature that causes a number of our dependencies to fail to install. — committed to apache/airflow by ashb 5 years ago
- NXDRIVE-1521: Stick with pip 18.1 As of now, pip 19.0.1 blocks the installation of PyInstaller on macOS. See that issue for details: https://github.com/pypa/pip/issues/6163 — committed to nuxeo/nuxeo-drive by deleted user 5 years ago
- NXDRIVE-1521: Stick with pip 18.1 As of now, pip 19.0.1 blocks the installation of PyInstaller. See that issue for details: https://github.com/pypa/pip/issues/6163 — committed to nuxeo/nuxeo-drive by deleted user 5 years ago
- NXDRIVE-1521: Stick with pip 18.1 As of now, pip 19.0.1 blocks the installation of PyInstaller. See that issue for details: https://github.com/pypa/pip/issues/6163 — committed to nuxeo/nuxeo-drive by deleted user 5 years ago
- [AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 (#4576) There is a bug or a new feature that causes a number of our dependencies to fail to install. — committed to apache/airflow by ashb 5 years ago
- Don't install pyinstaller and s3cmd in jobs that don't need it This will also probably circumvent the pip 19.0 pyinstaller [bug](https://github.com/pypa/pip/issues/6163) that hit us in all Travis job... — committed to hackaugusto/raiden by LefterisJP 5 years ago
- Don't install pyinstaller and s3cmd in jobs that don't need it This will also probably circumvent the pip 19.0 pyinstaller [bug](https://github.com/pypa/pip/issues/6163) that hit us in all Travis job... — committed to hackaugusto/raiden by LefterisJP 5 years ago
- Don't install pyinstaller and s3cmd in jobs that don't need it This will also probably circumvent the pip 19.0 pyinstaller [bug](https://github.com/pypa/pip/issues/6163) that hit us in all Travis job... — committed to hackaugusto/raiden by LefterisJP 5 years ago
- fix wine build: pyinstaller failed to install with new pip see pypa/pip#6163 — committed to spesmilo/electrum by SomberNight 5 years ago
- Issue #6163: Temporary workaround for legacy setup.py files — committed to ncoghlan/pip by ncoghlan 5 years ago
- Revert "[AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 (#4576)" This reverts commit 28b53e20736993dd3f513e449d44ab38986832d9. — committed to Fokko/incubator-airflow by deleted user 5 years ago
- PICARD-1456: Downgrade pip for macOS packaging. Work around for https://github.com/pypa/pip/issues/6163 — committed to phw/picard by phw 5 years ago
PyInstaller installs fine with
--no-use-pep517
.If it helps another example of this (via
apache-airflow
ispip install pendulum==1.4.4
fails, butpip install --no-use-pep517 pendulum==1.4.4
works.The stack trace we get is similar:
pip 19.0.2 has been released with a fix for this.
In case anyone’s wondering about the timeline, I expect we’ll be able to have a fix ready by the end of this week or early next week and make a subsequent bug-fix release of pip soon after that.
My proposal was actually that the opt-in for PEP 517 is specifying
build-system.build-backend
not the existence ofbuild-system
at all, and that between now and the 19.1 release,setuptools
would addbuild_meta_legacy
andpip
would use it as the default backend.I agree that in 19.1, probably if
pip
can’t findsetuptools.build_meta_legacy
, it should fall back to the old code path. That will give us minimal breaking changes while opting in the maximum number of people.Thanks for trawling the archives for us @cjerdonek. Stating my current understanding of the problem:
setup.py
files currently assume the current directory is onsys.path
when they run, creating weird bootstrapping issues as projects treat themselves as an install-time dependencysys.path
(it doesn’t say anything about backends)pip
19.0 considerspyproject.toml
without abuild-system
section to be equivalent to abuild-system
section that specifies thesetuptools
backendsetuptools
PEP 517 backend doesn’t currently add the current directory tosys.path
either (since getting away from point 1 above is considered a desirable future goal)pip
to less than 19.0, and explicitly setting the--no-use-pep517
option. However, folks only discover those after first discovering that upgradingpip
breaks their builds.From a functional perspective, I think the key change we want to introduce is that in the “
pyproject.toml
without abuild-system
section” case, then the directory containingsetup.py
should end up onsys.path
again. There are two main ways to handle that:pip
to do it. This has the unfortunate side effect of making the behaviour front-end dependent, and thus potentially see existing projects fail to install unless all frontends implement the same workaroundsetuptools
PEP 517 backend to do it, by inspectingpyproject.toml
for abuild-system
section, and injecting thesetup.py
directory intosys.path
if both the section and path entry are missing. A newpip
is still needed in that situation, since it has to specify a minimum version of setuptools that correctly handles the “nobuild-system
section” case.In the interests of getting things working for end users as quickly as possible, without painting ourselves into any unfortunate design corners, I’d actually propose a 3 step resolution:
pip
release (19.0.1?) now that adds the extrasys.path
entry for the “nobuild-system
entry” case. At this point, the compatibility issue should largely go away for end users.setuptools
release that handles the sys.path addition if the front end hasn’t already done so.pip
release, change the “nobuild-system
entry” case to drop the special casing, and instead set a stricter minimum version forsetuptools
.This proposal is based on the fact that I think the
setuptools
PEP 517 backend is the “right” place to handlesetup.py
backwards compatibility issues, but I also think that tweakingpip
directly is likely to be a much simpler change in the near term, and it’s a change that will contain the problem while the more architecturally preferable fix is worked on.I think it’s fairly common for people to import something from the current directory into a
setup.py
, and just generally treat things as ifsetup.py
is in$PWD
.I think it’s reasonable to push this responsibility onto
setuptools
, since that’s probably the only project that really needs it.@jce94, use pip<19 for now.
Even if we make pep517 opt-in, this is a behavior change that was not announced and will therefore break libraries even if they already opted in (see
PyInstaller
, for example). In the case where a library declares an opt-in to pep517 builds and imports things locally that it expects to find onsys.path
, there is no assumption being made by pip (because the library is explicit) but the library is still suddenly broken.In that case I really don’t see an alternative besides just including
cwd
in setuptools, because this is just broken. Unless the proposal is to tell people to go back and fix the releases they’ve cut in between pep517 and pip 19, which, if any user has those versions strictly pinned, may suddenly stop being installable, I really feel we should consider the impact of these decisions on the user experience. Based on this discussion and the current proposals some of these libraries will not be installable with new versions of pip + setuptools going forward using the defaults unless pep517 builds are explicitly disabled.This is pretty impactful if you’re just trying to install a package with the tooling that is provided to you by python, but actually it can’t install things somewhat randomly. I say this to draw the focus off of the technical aspects for a moment and onto the impact to the end user who may get frustrated with the tooling, the ecosystem, the libraries, or the language itself, because suddenly (and yes, only under specific circumstances), things they could install just fine now can’t be installed. I really do think we should close this gap in any solution that is implemented.
We currently use a failure stack to handle failed installations in pipenv and I’m adding
--no-use-pep517
when available to handle failures as a result of these changes. I’m not sure that will be intuitive to the average user, since it’s probably not even immediately clear what the cause of the problem is. I say this just to point out that we have a workaround, but it feels important to try and close this gap to help users out a bit on this one(edit: also big thanks to pganssle, cjerdonek, pfmoore, pradyunsg, ncoghlan, and everyone else who has been putting in a bunch of time and effort on this)
With both approaches operational on my local machine, I tested both #6210 and #6212 against the original problematic
PyInstaller==3.4
requirement.I don’t know whether the #6212 failure is a test setup problem, or if there’s actually a further problem in the setuptools pre-release, I only know that there’s further integration work to be done before we can be confident in that solution, whereas I’m confident in #6210 now - the only thing wrong with it is that it’s a horrible compatibility hack that we don’t want
pip
to have to carry forever.I would replace
setup.py
withpackage.json
and just add Python section for it instead of yet another package description format.At least then I can use
==1.x
version specifiers.This is not what I suggested, and it’s a bit of a digression. My point was that the problems that necessitated PEP 518 also exist more generally for all
setup.py
commands, and those are best solved by moving people away from invokingsetup.py
at all. No standard is necessary in order to do this in the same wayflit
does not need a PEP to add subcommands.So I skimmed a bunch of the emails, and I think you can at least skip ahead to August 29, where Nick seems to be suggesting there is consensus on leaving the source directory out of sys.path: https://mail.python.org/pipermail/distutils-sig/2017-August/031413.html (One by one, people were becoming convinced of Nathaniel’s arguments.)
However, in this same email linked above, Nick does say the following 😃
Here is a fuller paragraph from the email:
I don’t know yet if there are later emails that alter this summary, but there aren’t a whole lot of emails on the topic after.
That’s awesome! Regardless of the ultimate fix, this is a really nice example of the flexibility of the PEP 517 hook system 😃
For package maintainers looking to resolve this issue for your users: I’ve published a shim that implements the
sys.path
fix.https://pypi.org/project/setuptools-localimport/
Hopefully this can work as a stopgap so we can ponder how this should be moved forward without rushing into a solution, or unnecessarily slow down the adoption of pip 19.0 (which contains much more goodies than just PEP 517).
I don’t think this is a bug in pip/setuptools, from my reading of PEP 517’s Build Environment section it seems nothing about an environment should be assumed except that the dependencies declared in
pyproject.toml
are available.Also isn’t importing the package being installed from
setup.py
is a bad practice? There are much better ways of maintaining package version in one place, e.g. as described in this packaging guide from PyPA.As @cjerdonek mentioned in https://github.com/pypa/pip/issues/6175#issuecomment-456769285, could someone please check if
--no-use-pep517
fixes this for them?I suspect the cause of this issue is that build isolation or the PEP 517 code isn’t making sure that the root of the package directory is on the sys.path, because pandas has a versioneer.py sitting next to setup.py. I recall this coming up at some point, but I don’t remember off the top of my head what that discussion was. This might be considered an issue with the setuptools build backend instead of pip, or it might be the fault of pip’s isolation mechanism.
In a few hours. 😃
See the pinned issue.
The new version of setuptools, version
40.8.0
is now available with thebuild_meta:__legacy__
backend.I was unable to resolve this issue with the suggested workarounds while working with
pipenv
. Freezingpip
to18.1
in thePipfile
seems to have no effect aspipenv
keep forcing latest pip version. I can manually set pip to 18.1 but when I recreate the pipenv virtual environment Pipenv would upgrade to the latest pip no matter what…Any recommendations to make it stick?New PR #6229 that:
build-system
section inpyproject.toml
the initial requirement for automatically opting in to PEP 517 (deferring the “anypyproject.toml
file” opt-in to a later release)setup.py
I’m going to close the other 2 PRs in favour of that one, since it’s the minimal fix that should get things working again for end users, and doesn’t require any horrible hacks like #6210 or a new setuptools release like #6212
Working on #6210 has let me answer my own question from above about how hard it will be to address this at the
pip
level: the challenge is that the information about whether or not the source directory should be inserted assys.path[0]
needs to be tunnelled from the code that readspyproject.toml
through to the PEP 517 hook caller, and then from there into the actual in-process wrapper script.Those are doable without major architectural changes (a
pip._implicit.
prefix on the build backend name for the first part, and aPEP517_SYS_PATH_0
env var for the latter), but it means temporarily modifying the vendoredpep517
code until the proper fix insetuptools
is ready.I’ve created the
build_meta_legacy
backend in pypa/setuptools#1652. I would really prefer it ifpip
would switch to usingsetuptools.build_meta_legacy
as the default backend, but I think creating a built-in “legacy shim” backend is about as far as I’m comfortable going insetuptools
. I do not want to get stuck indefinitely supporting full “python setup.py install
emulation” in the mainsetuptools
PEP 517 backend.How would that work with
--no-binary :all:
?Personally, I agree with the current approach of relying on the existence of pyproject.toml file. The issues IMO stems from people using pyproject.toml for things other than packaging. The correct way out is therefore to push non-packaging tools to offer another way for configuration, so people can choose whether to use pyproject.toml or not.
success for me too
Ok, then that’s certainly an issue with the new PEP 517 code and I’m pretty sure the issue is just that the directory containing the project root hasn’t been added to
sys.path
. Maybe @pfmoore will have a better sense of if that should be pip’s responsbility or setuptools.