pants: `--use-first-matching-interpreter` shebang doesn't play nicely with macOS Python 3.7 install

It only would be picked back out of the cache if the BinaryPaths discovery was not re-run

This is false. The BinaryPaths discovery only finds a bootstrap interpreter to run the Pex CLI. The Pex CLI just needs some python 2.7 or python 3.5+. The interpreter used to build any individual PEX file is determined by the Pex CLI from interpreter constraints and platforms and the vagaries of what the Pex CLI happens to find on the system. This process is opaque to Pants today, although #10779 fixes that.

Noting another problem - this time on the Pants side - with using absolute path shebangs related to #10751. If a PEX file gets a statically determined python baked in it becomes a time bomb in the CAS. It will get plucked out to run next week when that interpreter is removed and go boom. The only remedy at that point will be to nuke the local lmdb.

It seems in general for local execution that relies on local binaries (not a problem for remote executiuon where images are static) Pants has this problem. We always need to check a cached binary exists and then re-run discovery if it does not before attempting to finally run the desired Process that uses that binary as argv0. In the case of a which style discovery process, this could be baked into the local command runner on the Rust side. In more complex cases like Pex interpreter discovery where the interpreter must be found, executed, and data extracted from the runtime environment to check applicability, it is alot easier to approach from the Python rule side of things.

In the case of PEX files, we could have a rule that uses a ./my.pex -c '' fallible Process to run the cached pex noop which should always exit 0 if an appropriate interpreter is still found. This would mean, once per pantsd lifecycle we’d have to no-op execute each cached pex tool using an absolute path shebang to verify it does not need to be rebuilt much like #10751 does.

True, but it’s a less aggressive breakage. But yeah, for right now the precise shebang is probably the way to go. I’ll put a change up.

Can the solution be as simple as creating a python3.7 symlink if needed?

I had tried creating that symlink in /usr/bin but macOS security disallowed that even when running as root.

The platforms thing alone doesn’t solve the issue, except on MacOS (assuming you’re not remoting to a MacOS cluster). But you could still get into trouble creating a pex with a full-path shebang on your linux desktop and then trying to use it remotely.

But then, that’s likely true even now, albeit maybe less often.

Normally, I’d be afraid it would break speculation, that that interpreter might be available locally but not in the remote environment. But that was always an issue, which we’ve worked around by marking building a Pex as being Platform specific. That is, the Pex you build locally isn’t expected to work on the remote machine to begin with.

I recommend Pants/Pex should not assume that the interpreter name is the same in both environments. They are different environments. As an example outside of Pants, Bazel’s “toolchain” concept models this and I believe a local toolchain and a remote toolchain could resolve different interpreter names (although worth looking to see how Bazel does Python toolchains).

https://github.com/pantsbuild/pex/blob/436dc554c1ac234e01529f9f275f7bd01130a51b/pex/interpreter.py#L197-L201 - pex assumes that it can form a binary name of pythonMAJOR.MINOR even if that is not actually the Python binary that it found when searching for an interpreter.