poetry: Running poetry install fails sometimes with new-installer

We’ve encountered this also running our builds on gitlab-ci, randomly builds will fail, not only running directly on the runner docker executor, also inside kaniko builds. The only solution has been to disable the new installer as documented above:

poetry config experimental.new-installer false

can we make the experimental not the default this has been driving me nuts for days thanks

I am seeing this issue using Poetry 1.1.7 in docker with the proposed workaround

RUN poetry env use system
RUN poetry config experimental.new-installer false
RUN poetry config virtualenvs.create false && poetry install

error installing

  Command ['/opt/hostedtoolcache/Python/3.9.6/x64/bin/python', '-m', 'pip', 'install', '--no-deps', '/home/runner/.cache/pypoetry/artifacts/b5/48/a1/d2b823337003d531d87cf0d503ef28bb579703a74d14ad24a88863d616/pickleshare-0.7.5-py2.py3-none-any.whl'] errored with the following return code 1, and output: 
  Processing /home/runner/.cache/pypoetry/artifacts/b5/48/a1/d2b823337003d531d87cf0d503ef28bb579703a74d14ad24a88863d616/pickleshare-0.7.5-py2.py3-none-any.whl
  ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages/packaging-20.9.dist-info'
  

We are also getting a lot of flakiness using the new installer in our CI builds. For example, those are errors from the three consecutive retries of the same build:

  Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--no-deps', '-U', '/root/.cache/pypoetry/artifacts/49/8a/46/de22c63977d20779dfe015492e797c0a4a785b6eea29f775267d233805/python-Levenshtein-0.12.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/49/8a/46/de22c63977d20779dfe015492e797c0a4a785b6eea29f775267d233805/python-Levenshtein-0.12.0.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-_izpglc5/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-_izpglc5/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-33zyawy4
           cwd: /tmp/pip-req-build-_izpglc5/
      Complete output (19 lines):
      running egg_info
      creating /tmp/pip-pip-egg-info-33zyawy4/python_Levenshtein.egg-info
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-req-build-_izpglc5/setup.py", line 46, in <module>
          """,
        File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 161, in setup
          return distutils.core.setup(**attrs)
        File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/usr/local/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 289, in run
          writer = ep.resolve()
        File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2468, in resolve
          module = __import__(self.module_name, fromlist=['__name__'], level=0)
      ModuleNotFoundError: No module named 'pbr'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
  You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
  

  at /pyroot/lib/python3.7/site-packages/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│ 
      1076│         return decode(output)
      1077│ 
      1078│     def execute(self, bin, *args, **kwargs):

  Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--no-deps', '-U', '/root/.cache/pypoetry/artifacts/49/8a/46/de22c63977d20779dfe015492e797c0a4a785b6eea29f775267d233805/python-Levenshtein-0.12.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/49/8a/46/de22c63977d20779dfe015492e797c0a4a785b6eea29f775267d233805/python-Levenshtein-0.12.0.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-d2hj8zxh/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-d2hj8zxh/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-buna0r_h
           cwd: /tmp/pip-req-build-d2hj8zxh/
      Complete output (3 lines):
      running egg_info
      creating /tmp/pip-pip-egg-info-buna0r_h/python_Levenshtein.egg-info
      error: [Errno 2] No such file or directory: '/pyroot/lib/python3.7/site-packages/~br-5.5.1.dist-info/METADATA'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
  You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
  

  at /pyroot/lib/python3.7/site-packages/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│ 
      1076│         return decode(output)
      1077│ 
      1078│     def execute(self, bin, *args, **kwargs):

  Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--no-deps', '-U', '/root/.cache/pypoetry/artifacts/e5/29/47/442c47c9e875153abd2315ffc8b40e50a84dc8ac5011d9bf1530673205/uvicorn-0.12.2-py3-none-any.whl'] errored with the following return code 2, and output: 
  ERROR: Exception:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
      status = self.run(options, args)
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
      return func(self, options, args)
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 278, in run
      session = self.get_default_session(options)
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 83, in get_default_session
      self._session = self.enter_context(self._build_session(options))
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 100, in _build_session
      index_urls=self._get_index_urls(options),
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/session.py", line 249, in __init__
      self.headers["User-Agent"] = user_agent()
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/network/session.py", line 159, in user_agent
      setuptools_version = get_installed_version("setuptools")
    File "/usr/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 634, in get_installed_version
      working_set = pkg_resources.WorkingSet()
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 567, in __init__
      self.add_entry(entry)
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 623, in add_entry
      for dist in find_distributions(entry, True):
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2065, in find_on_path
      for dist in factory(fullpath):
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2127, in distributions_from_metadata
      if len(os.listdir(path)) == 0:
  FileNotFoundError: [Errno 2] No such file or directory: '/pyroot/lib/python3.7/site-packages/~lackclient-2.9.3.dist-info'
  WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
  You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
  

  at /pyroot/lib/python3.7/site-packages/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│ 
      1076│         return decode(output)
      1077│ 
      1078│     def execute(self, bin, *args, **kwargs):

Added a step to the CI to upload the artifacts on failure and that worked for getting a copy on another failed run.

Failure this time was on CairoSVG-2.4.2-py3-none-any.whl.

The sha256 from PyPi is 9cb1df7e9bc60f75fb87f67940a8fb805aad544337a67a40b67c05cfe33711a2.

The sha256 of the file from the artifacts on that build matches PyPi:

❯ sha256sum CairoSVG-2.4.2-py3-none-any.whl 
9cb1df7e9bc60f75fb87f67940a8fb805aad544337a67a40b67c05cfe33711a2  CairoSVG-2.4.2-py3-none-any.whl