pex: pex fails with InvalidSpecifier if any dist has a bad version specifier
After upgrading from 2.1.100 to 2.1.151 ~and so pulling https://github.com/pantsbuild/pex/pull/2264~ pex started failing to execute locally with the following error:
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "~/.pex/unzipped_pexes/003cdfcf7d5635cb324a06f8b49939c255f2add1/__main__.py", line 108, in <module>
bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
File "pex/pex_bootstrapper.py", line 627, in bootstrap_pex
pex.PEX(entry_point).execute()
File "pex/pex.py", line 542, in execute
self.activate()
File "pex/pex.py", line 242, in activate
self.patch_sys()
File "pex/pex.py", line 445, in patch_sys
new_sys_path, new_sys_path_importer_cache, new_sys_modules = self.minimum_sys(inherit_path)
File "pex/pex.py", line 415, in minimum_sys
sys_path, sys_path_importer_cache = self.minimum_sys_path(isolated_sys_path, inherit_path)
File "pex/pex.py", line 351, in minimum_sys_path
site_distributions.update(all_distribution_paths(path_element))
File "pex/pex.py", line 345, in all_distribution_paths
locations = set(dist.location for dist in find_distributions([path]))
File "pex/pex.py", line 345, in <genexpr>
locations = set(dist.location for dist in find_distributions([path]))
File "pex/dist_metadata.py", line 1004, in find_distributions
location=location, metadata=DistMetadata.from_metadata_files(metadata_files)
File "pex/dist_metadata.py", line 758, in from_metadata_files
requires_python=requires_python(metadata_files),
File "pex/dist_metadata.py", line 516, in requires_python
return SpecifierSet(python_requirement)
File "pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py", line 714, in __init__
parsed.add(Specifier(specifier))
File "pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py", line 245, in __init__
raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
pex.vendor._vendored.packaging_23_1.packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=3.5.*'
We managed to trace this back to apache_libcloud-3.2.0.egg-info/PKG-INFO which has the following line:
Requires-Python: >=3.5.*, <4
Although this seems to be a bad specifier , some components presumably accept it as the package was working in that state for years, but not the packaging stuff used by pex.
Now this was a bug in libcloud [1], but I think pex should not fail entirely if some package has this in its version info. ~This was exposed by #2264 because before that this PKG-INFO file was not parsed at all.~
In any case libcloud has been fixed years ago apparently but of course not every transitive dependency has necessarily been updated to use a fixed version, etc.
About this issue
- Original URL
- State: closed
- Created 7 months ago
- Comments: 24 (16 by maintainers)
Commits related to this issue
- Remove Pex runtime scrubbing dist discovery. This was a relic of the days of egg zips; and the `dist.location` data is now fully redundant with the `sys.path` element input to find the dist in the 1s... — committed to jsirois/pex by jsirois 7 months ago
- Remove Pex runtime scrubbing dist discovery. (#2290) This was a relic of the days of egg zips; and the `dist.location` data is now fully redundant with the `sys.path` element input to find the dist... — committed to pex-tool/pex by jsirois 7 months ago
Ok, using the same Dockerfile from above, the repro case is now fixed:
The fix will go out in 2.1.153: #2291
FWIW this bug will be fixed and released in under 8 hours start to finish. I would guess you don’t get that good a response time in many OSS projects; especially those maintained gratis. I am not payed for this or any other software work.
Ah, I guess I did apologize - but that was a bit tongue in cheek.