pex: pex vendoring does not (always) isolate itself from a colliding setuptools install in site-packages
Having issues creating pex files in an image based on databricksruntime/python-virtualenv.
Dockerfile:
FROM databricksruntime/python-virtualenv:latest
RUN apt-get update \
&& apt-get install -y --allow-unauthenticated apt-transport-https ca-certificates curl gnupg2 software-properties-common curl software-properties-common build-essential
# Install & Bootstrap Python 3.7
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt update
RUN apt install --yes python3.7 python3.7-dev
RUN virtualenv -p python3.7 --system-site-packages /databricks/python3
RUN /databricks/python3/bin/pip3.7 install pex setuptools==45.2.0
ENV PATH="$PATH:/databricks/python3/bin"
Command run in container:
pex --interpreter-constraint='CPython>=3.7' boto3 -o boto3.pex -m boto3
Portion of failure StackTrace:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-kh23prp9/setup.py", line 59, in <module>
'Programming Language :: Python :: 3.8',
File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/__init__.py", line 169, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/wheel/wheel/bdist_wheel.py", line 228, in run
self.run_command('install')
File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/install.py", line 65, in run
return orig.install.run(self)
File "/usr/lib/python3.7/distutils/command/install.py", line 601, in run
self.run_command(cmd_name)
File "/usr/lib/python3.7/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/install_scripts.py", line 22, in run
import setuptools.command.easy_install as ei # vendor:skip
File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/command/easy_install.py", line 69, in <module>
from setuptools.sandbox import run_setup # vendor:skip
File "/root/.pex/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49/.deps/setuptools/setuptools/sandbox.py", line 24, in <module>
import pkg_resources.py31compat # vendor:skip
ModuleNotFoundError: No module named 'pkg_resources.py31compat'
----------------------------------------
ERROR: Failed building wheel for boto3
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (11 by maintainers)
Commits related to this issue
- Improve `PythonInterpreter` venv support. This fixes binary canonicalization to handle virtual environments created with virtualenv instead of pyvenv. It also adds support for resolving the base inte... — committed to jsirois/pex by jsirois 4 years ago
- Escape venvs unless PEX_INHERIT_PATH is requested. Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of ... — committed to jsirois/pex by jsirois 4 years ago
- Escape venvs unless PEX_INHERIT_PATH is requested. Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of ... — committed to jsirois/pex by jsirois 4 years ago
- Improve `PythonInterpreter` venv support. (#1129) This fixes binary canonicalization to handle virtual environments created with virtualenv instead of pyvenv. It also adds support for resolving the... — committed to pex-tool/pex by jsirois 4 years ago
- Escape venvs unless PEX_INHERIT_PATH is requested. Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breaking out of ... — committed to jsirois/pex by jsirois 4 years ago
- Escape venvs unless PEX_INHERIT_PATH is requested. (#1130) Virtual environments created with --system-site-packages can break PEX sys.path scrubbing. Avoid this class of environment leaks by breakin... — committed to pex-tool/pex by jsirois 4 years ago
It looks like one way to solve this is to invert how we calculate which
sys.pathentries to scrub. Instead of calculating site-libraries, user-site-libraries and extras to scrub: https://github.com/pantsbuild/pex/blob/0a29161e3b24a1b7a884ab3acd5f2f411bac7336/pex/pex.py#L309-L315 We could calculate just the core libraries and scrub everything else via:That solves the case above and seems as robust as you can possibly be since the answer is delegated to the python interpreter in question.
As requested: @stuhood