pex: `pex3 lock create` does not work with VCS requirements as input
This works:
pex 'ansicolors @ git+https://github.com/jonathaneunice/colors.git@c965f5b9103c5bd32a1572adb8024ebe83278fb0'
But this fails:
❯ pex3 lock create 'ansicolors @ git+https://github.com/jonathaneunice/colors.git@c965f5b9103c5bd32a1572adb8024ebe83278fb0'
Traceback (most recent call last):
File "/Users/ericarellano/.local/bin/pex3", line 8, in <module>
sys.exit(main())
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/cli/pex.py", line 29, in main
result = catch(command.run)
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/commands/command.py", line 130, in catch
return func(*args, **kwargs)
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/cli/command.py", line 71, in run
return subcommand_func(self)
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/cli/commands/lock.py", line 199, in _create
create(
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/cli/commands/lockfile/__init__.py", line 113, in create
downloaded = resolver.download(
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/resolver.py", line 1097, in download
build_requests, download_results = _download_internal(
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/resolver.py", line 1002, in _download_internal
download_results = download_request.download_distributions(
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/resolver.py", line 134, in download_distributions
return list(
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/jobs.py", line 510, in execute_parallel
yield spawn_result.spawned_job.await_result()
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/jobs.py", line 214, in await_result
job.wait()
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/jobs.py", line 73, in wait
self._check_returncode(stderr)
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/pip.py", line 519, in _check_returncode
result = analyzer.analyze(line)
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/pip.py", line 442, in analyze
project_name_and_version, partial_artifact = self._extract_resolve_data(
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/pip.py", line 425, in _extract_resolve_data
pin = Pin.canonicalize(ProjectNameAndVersion.from_filename(urlparse.urlparse(url).path))
File "/Users/ericarellano/.local/pipx/venvs/pex/lib/python3.9/site-packages/pex/dist_metadata.py", line 237, in from_filename
raise UnrecognizedDistributionFormat(
pex.dist_metadata.UnrecognizedDistributionFormat: The distribution at path '/jonathaneunice/colors.git@c965f5b9103c5bd32a1572adb8024ebe83278fb0' does not have a file name matching known sdist or wheel file name formats.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (14 by maintainers)
Commits related to this issue
- Reject VCS requirements in locks. (#1563) The current pex lock infrastructure cannot handle VCS requirements in the same way it can't handle local project requirements. Unlike local project require... — committed to pex-tool/pex by jsirois 2 years ago
- Add support for locking VCS requirements. (#1687) This supports all forms of VCS requirements Pip supports as direct or transitive requirements. VCS source archives fetched by Pip are sha256 hashe... — committed to pex-tool/pex by jsirois 2 years ago
Alright, it turns out option 4 works, VCS urls of any form can be anywhere in the dependency graph and things work. It turns out I could just rely on Pip to handle vcs downloading and post process the resulting VCS zips it creates to get a hash of the contained source tree for a reproducible strong fingerprint regardless of commit id or no. Some cleanup work left, but end to end create VCS locks and then consume them is working for all 4 VCS systems supported by Pip: https://github.com/jsirois/pex/tree/issues/1556/end-run
We’ll see how tricky this is. As a 1st cut I’ll try just supporting git and maybe local projects that are git controlled as well.
I would like option 3:
The VCS requirements we have go to smaller repos with fewer deps. The sub repos do not have any VCS requirement. So I think raising an error for any internal VCS nodes, or requiring commit IDs for them would be fine.
At least in StackStorm’s code, targeting a branch but locking to a commit is exactly what I’d like to do, so I would prefer that work for top-level VCS dependencies if possible.
@cognifloyd in your example, you have VCS requirements that are, on the face of it, mutable; i.e.: they do not reference commit ids, just branches or tags. Pex can handle this and pre-clone repos, grab commit ids and then hand off to Pip as a local project directory, writing down the original VCS url in the lock file and using the commit id for the hash.
That, though, falls apart if there are interior nodes in the dependency graph that likewise use VCS urls without commit ids. All Pex can do with these is find out about them after Pip has run and done the lock resolve. At that point Pex could clone the newly discovered interior node VCS urls to find out commit ids, but that is broken since the commit id for a branch may have changed in the time between the pip run and the lock post-processing.
It seems to me there are 3 choices here:
Do you have opinions here?