pants: 2.3.0rc0 - ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2

Hi all,

I just upgraded from 2.3.0.dev3 to 2.3.0rc0 and running ./pants version wouldn’t finish without an error. I can successfully compile and test with 2.3.0.dev1 and 2.3.0.dev3 (didn’t test dev2) - but rc0 fails before it finishes installing/setting up.

Note: It DOES work on Mac, but fails on Debian.

Mac Environment (works)

macOS Big Sur (11.2.1)

Python 3.7.9 (default, Jan 22 2021, 16:36:49) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Python 3.9.1 (default, Jan 22 2021, 16:49:02) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Linux Environment (fails)

Debian GNU/Linux 10 (buster)
Linux 4.19.0-10-amd64

Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux

Python 3.9.1 (default, Feb 3 2021, 20:42:26)
[GCC 8.3.0] on linux

Stacktrace:

./pants version

10:33:48.67 [INFO] initializing pantsd...
10:33:49.60 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.7.3)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2.
stdout:

stderr:
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: syntax error near unexpected token `('
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: `    venv_dir = os.path.abspath(os.path.dirname(__file__))'

Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 154, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 124, in prepare
    self._initialize(options_fingerprint, options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 99, in _initialize
    raise e
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 85, in _initialize
    options_bootstrapper, raise_=True
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/options_initializer.py", line 108, in build_config_and_options
    build_config = _initialize_build_configuration(self._plugin_resolver, options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/options_initializer.py", line 43, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper)
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/plugin_resolver.py", line 109, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/init/plugin_resolver.py", line 130, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [self._interpreter_constraints])[
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 592, in product_request
    self._raise_on_error([t for _, t in throws])
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins (CPython==3.7.3)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/parallels/.cache/pants/setup/bootstrap-Linux-x86_64/2.3.0rc0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Extracting plugin locations' failed with exit code 2.
stdout:

stderr:
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: syntax error near unexpected token `('
/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/pex: line 7: `    venv_dir = os.path.abspath(os.path.dirname(__file__))'

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (21 by maintainers)

Commits related to this issue

Upgrade Pex to 2.1.32. This fixes venv script shebangs being too long for some Linux systems. Fixes #11597 [ci skip-rust] [ci skip-build-wheels] — committed to jsirois/pants by jsirois 3 years ago
Upgrade Pex to 2.1.32. (#11604) This fixes venv script shebangs being too long for some Linux systems. Fixes #11597 — committed to pantsbuild/pants by jsirois 3 years ago
Upgrade Pex to 2.1.32. (#11604) This fixes venv script shebangs being too long for some Linux systems. Fixes #11597 # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust]... — committed to jsirois/pants by jsirois 3 years ago
Upgrade Pex to 2.1.32. (#11604) (#11606) This fixes venv script shebangs being too long for some Linux systems. Fixes #11597 — committed to pantsbuild/pants by jsirois 3 years ago

Most upvoted comments

@jsirois Thanks! Can confirm this works for me now.

I should also note that in plugin_resolver.py, the following code meant I completely bypassed the venv workflow (and thereby, bypassed the shebang problem) in rc1:

if not requirements:
    return ResolvedPluginDistributions()

To sanity check, I commented that code out, and can confirm that I’m now getting the named_caches/pex_root/venvs/short/{8ByteHash} path and that also succeeds.

sureshjoshi on Feb 26, 2021

Ah, so in the never ending - read closer dummy department - I read closer and got less dumb: “syntax error near unexpected token `('” - That’s bash speaking and not Python AFAICT. Ruminating on that…

jsirois on Feb 24, 2021

https://github.com/pantsbuild/pex/issues/1252

jsirois on Feb 25, 2021

I’ll spin up a 2.3.0rc1 as soon as #11604 lands with the Pex upgrade.

jsirois on Feb 25, 2021

Ah yes. Thanks. Not a known Python issue - just a known shebang issue regardless of interpreter. I’ll fork a Pex issue, fix over there and upgrade Pants, etc.

Thanks again for digging with me @sureshjoshi .

jsirois on Feb 25, 2021

FWIW: This shebang thing seems to be a known Python issue after looking for it

https://www.in-ulm.de/~mascheck/various/shebang/ https://github.com/pypa/pip/issues/1773 https://github.com/pypa/virtualenv/issues/596 https://github.com/pypa/virtualenv/issues/997

It also looks like the latest pex re-wrote a lot of code around this functionality with the venvs __main__.py file. Not sure if it’s related - since the shebang at the top is the same, but there seems to be code to re-exec if python isn’t in use: https://github.com/pantsbuild/pex/compare/v2.1.30...v2.1.31#diff-44e8fd52cd3a63a2d3f2fc516418c2aa55e8ee96f6eb82b2186765a8b853922f

Is there some way that code be added to create a pex_root symlink to the appropriate venv? That could chomp the length in half probably.

As I’m not familiar with the codebase, I’m not really sure why the venvs folders are so nested (some pex hashes I guess along the way) - so I don’t know if there are expected to be multiple venvs in the same folders which would necessitate some clever symlink naming in my example (which could just be a pain to do and maintain).

What really sucks is that this appears to be an OS-specific version, and if the Linux distro in use increases the specified length, this is basically a non-issue.

sureshjoshi on Feb 25, 2021

@jsirois Weird question… Could we be exceeding the shebang line limit?

It’s 155 characters in my example, I put a symlink to my home dir, used that instead, and running ./pex now works

Symlink:

ln -s /home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python3.7 ~/pppython

In __main__.py:

From:

#!/home/parallels/.cache/pants/named_caches/pex_root/venvs/be44469c49a60b73e5eddf92e8aeeff77eaf7943/800ae8ef047bd783543e8282c22bfdbee7b7fca8/bin/python3.7 -sE

To:

#! /home/parallels/pppython -sE

./pants version succeeds

sureshjoshi on Feb 24, 2021

@stuhood Attached

pex.zip

pants.toml - created an empty project, no special pip resolvers (that fails too though), no other fields.

[GLOBAL]
pants_version = "2.3.0rc0"

@Eric-Arellano No worries, there is no particular reason for me to use rc0 specifically.

I’m using 2.3.x because of the venv additions in the pex tool, but every now and again, I just upgrade to the latest dev versions to keep an eye on regressions that might burn me when I have less free time 😃

dev3 works perfectly fine for me - so just wanted to post this issue as a flag for anyone else who might run into it if they’re doing what I am too.

sureshjoshi on Feb 24, 2021