pants: ModuleNotFoundError: No module named 'pex'

Describe the bug When running pants in our CI system, we can have a container that has multiple clones of our repository. At various points, they will be running goals like ./pants export or ./pants run. At some point in the chain, we see errors like this

10:42:12.54 [INFO] Initialization options changed: reinitializing scheduler...
10:42:13.10 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/.pex/unzipped_pexes/7907cee4a80399696d31dbee748702a492b91a80/__main__.py", line 102, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'

The only way to remedy this seems to be clearing out ~/.pex and trying the goal again.

Pants version 2.13.0

OS Ubuntu 20.04

Additional info We’ve tried adding some steps to manage concurrency in the system and automatically wipe ~/.pex when this error is detected, but that’s unreliable to implement everywhere. Some initially debugging was kindly provided by @jsirois who filed https://github.com/pantsbuild/pants/issues/17176 to resolve.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 53 (35 by maintainers)

Commits related to this issue

Fix incomplete cherry-pick of #17177. That cherry-pick was done in #17200 and missed a crucial line which led to `~/.cache/pants/named_caches/pex_root` being nuked on any run of `./pants export` inst... — committed to jsirois/pants by jsirois 2 years ago
Fix incomplete cherry-pick of #17177. (#17312) That cherry-pick was done in #17200 and missed a crucial line which led to `~/.cache/pants/named_caches/pex_root` being nuked on any run of `./pants e... — committed to pantsbuild/pants by jsirois 2 years ago
Improve the cache nuking recommendation. (#20385) Pants fails in hard-to-debug ways if a cache is partially deleted. This recommendation ensures that the cache nuking is atomic from Pants's perspe... — committed to pantsbuild/pants by benjyw 6 months ago

Most upvoted comments

Great news! Looks like our builds that we’re regularly failing were successful last night. I believe you are good to proceed with your release process.

engnatha on Oct 26, 2022

@Eric-Arellano OK. Thanks, that’s good to know.

I’ve got https://github.com/pantsbuild/pex/pull/1961 spun up to hopefully make errors more clear for the #16778 style case, although that change fixes nothing afaict since I still can’t see the underlying bug for that case.

For the 3 cases I extracted from your failed CI runs so far (more work to do looking at the last run you posted in https://github.com/pantsbuild/pants/actions/runs/3283455197/jobs/5409915111) that change may or may not help make the underlying issue more clear. I still need to do analysis on all those backtraces against 2.1.108 code.

jsirois on Oct 20, 2022

@Eric-Arellano I’ve filed https://github.com/pantsbuild/pex/issues/1955, https://github.com/pantsbuild/pex/issues/1956 and https://github.com/pantsbuild/pex/issues/1957 for the 3 variants I found using your links. That’s crazy though to see so many different types of missing file errors at once. I really have no clue what’s up here. Have you seen this sort of thing regularly or just with that PR or ???.

jsirois on Oct 19, 2022

From my phone, but afaict no. The issues there are all named caches related. The issue @engnatha experienced was completely different where Pants was foisted on it’s own petard when running ~rm -rf on ~/.pex during export - decidedly not in named caches. @benjyw fixed the foisting issue.

jsirois on Oct 19, 2022