pants: Lockfile generation fails with KeyError type exception

Describe the bug Running ./pants generate-lockfiles --resolve=python-default eventually crashes with a KeyError type exception.

Full stack trace:

 ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/__main__.py", line 103, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex_bootstrapper.py", line 601, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 528, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 435, in _wrap_coverage
    return runner(*args)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 466, in _wrap_profiling
    return runner(*args)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 547, in _execute
    return self.execute_script(self._pex_info_overrides.script)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 679, in execute_script
    return self.execute_entry(dist_entry_point.entry_point)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 737, in execute_entry
    return self.execute_entry_point(entry_point)
  File "/home/johan/.cache/pants/named_caches/pex_root/unzipped_pexes/dfe5a70dd34dac2e7fbc472cbc8a8b4ecc5a9c1a/.bootstrap/pex/pex.py", line 755, in execute_entry_point
    return runner()
  File "/home/johan/.cache/pants/named_caches/pex_root/installed_wheels/0a7db6e5ee325f3fa984fc8c15300400fa1c62554af22dd1d811f14f9ed52798/pex-2.1.99-py2.py3-none-any.whl/pex/cli/pex.py", line 32, in main
    result = catch(command.run)
  File "/home/johan/.cache/pants/named_caches/pex_root/installed_wheels/0a7db6e5ee325f3fa984fc8c15300400fa1c62554af22dd1d811f14f9ed52798/pex-2.1.99-py2.py3-none-any.whl/pex/result.py", line 103, in catch
    return func(*args, **kwargs)
  File "/home/johan/.cache/pants/named_caches/pex_root/installed_wheels/0a7db6e5ee325f3fa984fc8c15300400fa1c62554af22dd1d811f14f9ed52798/pex-2.1.99-py2.py3-none-any.whl/pex/cli/command.py", line 84, in run
    return subcommand_func(self)
  File "/home/johan/.cache/pants/named_caches/pex_root/installed_wheels/0a7db6e5ee325f3fa984fc8c15300400fa1c62554af22dd1d811f14f9ed52798/pex-2.1.99-py2.py3-none-any.whl/pex/cli/commands/lock.py", line 376, in _create
    create(
  File "/home/johan/.cache/pants/named_caches/pex_root/installed_wheels/0a7db6e5ee325f3fa984fc8c15300400fa1c62554af22dd1d811f14f9ed52798/pex-2.1.99-py2.py3-none-any.whl/pex/resolve/lockfile/create.py", line 294, in create
    create_lock_download_manager = CreateLockDownloadManager.create(
  File "/home/johan/.cache/pants/named_caches/pex_root/installed_wheels/0a7db6e5ee325f3fa984fc8c15300400fa1c62554af22dd1d811f14f9ed52798/pex-2.1.99-py2.py3-none-any.whl/pex/resolve/lockfile/create.py", line 81, in create
    artifact = file_artifacts_by_filename.get(f) or source_artifacts_by_pin[pin]
KeyError: Pin(project_name=ProjectName(raw='fonttools', normalized='fonttools'), version=Version(raw='4.34.4', normalized='4.34.4'))

Our current workaround is to temporarily switch to version 2.12.1rc2 where lockfile generation still works.

Pants version 2.13.0rc0

OS MacOS, Linux (WSL)

Additional info We’re using an alternative index as opposed to the default pypi.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 20 (14 by maintainers)

Most upvoted comments

And it turns out in 2.1.99 with the lack of os_name == "nt" handling fixed y 2.1.100, the Pex lock analyzer’s analysis of Pip logging is foiled leading to the strange error seen here. So this is all satisfactorily explained in the Pants context and 2.13.0rc1 will have your fix via Pex 2.1.102, but I’ll still file a Pex issue - I think the Pip log analysis can be made more robust here for the future.

@johanols I’ll leave this open but I’d like to close as soon as we release 2.13.0rc1 and you can try upgrading to that to see if it fixes this issue.

jsirois on Aug 12, 2022

@Eric-Arellano With 2.13.0rc1 I no longer have any issues generating the lockfile! It’s also way faster 😃 Thank you all for your superb support!

johanols on Aug 16, 2022

Ok, I can make 2.1.99 pass by eliminating both --target-system flags, which does put this back in the wheel-house of the 2.1.100 fix, but I still can’t grok how yet.

jsirois on Aug 11, 2022

Alright - thank you. That gave me enough to repro. I’m drilling in presently and will probably file a bug over in Pex and link that back here.

jsirois on Aug 11, 2022

Thank you @Eric-Arellano for the swift response.

Is this the output you’re talking about:

[
    "/home/my-user/project/dist/export/python/virtualenvs/python-default/3.8.13/bin/python",
    "./pex",
    "lock",
    "create",
    "--tmpdir",
    ".tmp",
    "--python-path",
    "/home/my-user/project/dist/export/python/virtualenvs/python-default/3.8.13/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/my-user/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/my-user/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin",
    "--output=lock.json",
    "--no-emit-warnings",
    "--style=universal",
    "--resolver-version",
    "pip-2020-resolver",
    "--target-system",
    "linux",
    "--target-system",
    "mac",
    "--indent=2",
    "-r",
    "__pip_args.txt",
    "--no-pypi",
    "--index=https://our.pypi.server/simple",
    "--manylinux",
    "manylinux2014",
    "--interpreter-constraint",
    "CPython==3.8.*",
    "altair>=4.1.0",
    "attrs>=20.3.0",
    "azure_storage_blob>=12.11.0",
    "cachetools>=4.2.4",
    "cloudpickle>=2.0.0",
    "dask==2022.7.0",
    "dill>=0.3.2",
    "distributed==2022.7.0",
    "einops==0.3.0",
    "elastic-apm>=6.10.1",
    "fastapi[all]>=0.79.0",
    "freezegun>=1.1.0",
    "graphviz>=0.19.1",
    "gunicorn>=20.1.0",
    "ipywidgets==7.6.5",
    "jinja2>=2.11.3",
    "jupyterlab==4.0.0a26",
    "kartothek==5.3.0",
    "locust>=2.10.1",
    "matplotlib>=3.5.1",
    "matplotlib_inline>=0.1.3",
    "mock>=4.0.3",
    "more_itertools>=8.12.0",
    "nbconvert>=6.0.7",
    "nbformat>=5.1.3",
    "networkx>=2.4",
    "numba>=0.53.1",
    "numpy==1.22.3",
    "onnx==1.11.0",
    "onnxruntime==1.11.0",
    "orjson",
    "pandas==1.4.3",
    "pandas_aggregator==1.4.1",
    "pandas_interface==0.19.0",
    "protobuf==3.19.4",
    "py-spy>=0.3.12",
    "pyarrow==6.0.1",
    "pydantic>=1.9.0",
    "pytest_mock>=3.5.1",
    "pytorch_lightning==1.6.1",
    "scikit-learn==1.1.1",
    "scipy==1.8.1",
    "setuptools>=59.5.0",
    "simplekv==0.14.1",
    "storefact==0.10.0",
    "streamlit>=1.8.0",
    "structlog==21.5.0",
    "tensorboard==2.8.0",
    "toolz>=0.11.2",
    "torch==1.11.0",
    "torchmetrics==0.7.3",
    "types-cachetools",
    "types-freezegun",
    "types-requests",
    "types_mock",
    "typing_extensions"
]

johanols on Aug 10, 2022