poetry: Frequent intermittent connection failures

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Debian GNU/Linux 10 (buster)
Poetry version: 1.1.3
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/maroux/5bfc38375acd0e1c1a710fc0e62b8e93

Issue

Environment: Docker debian slim buster image running in Google Cloud Platform Cloud Build Installation: poetry install -vvv --no-root --no-dev Outcome: Fails with networking error intermittently, but frequently enough that upgrade to poetry v1.1 is blocked on solving this issue. Notes: Older version (v1.0.9) works fine but is slow because of old installer. Verbose logs:

16:05:17 Step #2 - "build": Skipping virtualenv creation, as specified in config file.
16:05:17 Step #2 - "build": Installing dependencies from lock file
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build": Finding the necessary packages for the current system
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build": Package operations: 17 installs, 1 update, 0 removals, 11 skipped
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   ? Installing certifi (2020.6.20): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing chardet (3.0.4): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing idna (2.10): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing markupsafe (1.1.1)
16:05:17 Step #2 - "build":   ? Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing pytz (2020.1): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing six (1.15.0): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing urllib3 (1.25.10): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing alabaster (0.7.12)
16:05:17 Step #2 - "build":   ? Installing babel (2.8.0)
16:05:17 Step #2 - "build":   ? Updating colorama (0.4.4 -> 0.4.3): Skipped for the following reason: Not needed for the current environment
16:05:17 Step #2 - "build":   ? Installing docutils (0.16): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing imagesize (1.2.0)
16:05:17 Step #2 - "build":   ? Installing jinja2 (2.11.2)
16:05:17 Step #2 - "build":   ? Installing packaging (20.4): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Updating pygments (2.7.1 -> 2.6.1)
16:05:17 Step #2 - "build":   ? Installing requests (2.24.0): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing snowballstemmer (2.0.0)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-applehelp (1.0.1)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-devhelp (1.0.1)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-htmlhelp (1.0.2)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-jsmath (1.0.1)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-qthelp (1.0.2)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-serializinghtml (1.1.3)
16:05:17 Step #2 - "build":   ? Installing commonmark (0.9.1)
16:05:17 Step #2 - "build":   ? Installing sphinx (3.1.2)
16:05:17 Step #2 - "build": Retrying HTTP request in 0.5 seconds.
16:05:17 Step #2 - "build":   ? Installing recommonmark (0.5.0)
16:05:17 Step #2 - "build":   ? Installing sphinx-autodoc-typehints (1.11.0)
16:05:17 Step #2 - "build":   ? Installing sphinx-rtd-theme (0.4.3)
16:05:17 Step #2 - "build": Retrying HTTP request in 0.5 seconds.
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   Stack trace:
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   13  ~/.poetry/lib/poetry/installation/executor.py:199 in _execute_operation
16:05:17 Step #2 - "build":        197? 
16:05:17 Step #2 - "build":        198?             try:
16:05:17 Step #2 - "build":      ? 199?                 result = self._do_execute_operation(operation)
16:05:17 Step #2 - "build":        200?             except EnvCommandError as e:
16:05:17 Step #2 - "build":        201?                 if e.e.returncode == -2:
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   12  ~/.poetry/lib/poetry/installation/executor.py:273 in _do_execute_operation
16:05:17 Step #2 - "build":        271?             return 0
16:05:17 Step #2 - "build":        272? 
16:05:17 Step #2 - "build":      ? 273?         result = getattr(self, "_execute_{}".format(method))(operation)
16:05:17 Step #2 - "build":        274? 
16:05:17 Step #2 - "build":        275?         if result != 0:
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   11  ~/.poetry/lib/poetry/installation/executor.py:408 in _execute_install
16:05:17 Step #2 - "build":        406? 
16:05:17 Step #2 - "build":        407?     def _execute_install(self, operation):  # type: (Install) -> None
16:05:17 Step #2 - "build":      ? 408?         return self._install(operation)
16:05:17 Step #2 - "build":        409? 
16:05:17 Step #2 - "build":        410?     def _execute_update(self, operation):  # type: (Update) -> None
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   10  ~/.poetry/lib/poetry/installation/executor.py:434 in _install
16:05:17 Step #2 - "build":        432?             archive = self._download_link(operation, Link(package.source_url))
16:05:17 Step #2 - "build":        433?         else:
16:05:17 Step #2 - "build":      ? 434?             archive = self._download(operation)
16:05:17 Step #2 - "build":        435? 
16:05:17 Step #2 - "build":        436?         operation_message = self.get_operation_message(operation)
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    9  ~/.poetry/lib/poetry/installation/executor.py:575 in _download
16:05:17 Step #2 - "build":        573? 
16:05:17 Step #2 - "build":        574?     def _download(self, operation):  # type: (Operation) -> Path
16:05:17 Step #2 - "build":      ? 575?         link = self._chooser.choose_for(operation.package)
16:05:17 Step #2 - "build":        576? 
16:05:17 Step #2 - "build":        577?         return self._download_link(operation, link)
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    8  ~/.poetry/lib/poetry/installation/chooser.py:60 in choose_for
16:05:17 Step #2 - "build":         58?         """
16:05:17 Step #2 - "build":         59?         links = []
16:05:17 Step #2 - "build":      ?  60?         for link in self._get_links(package):
16:05:17 Step #2 - "build":         61?             if link.is_wheel and not Wheel(link.filename).is_supported_by_environment(
16:05:17 Step #2 - "build":         62?                 self._env
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    7  ~/.poetry/lib/poetry/installation/chooser.py:94 in _get_links
16:05:17 Step #2 - "build":         92?             repository = self._pool.repository(package.source_reference)
16:05:17 Step #2 - "build":         93? 
16:05:17 Step #2 - "build":      ?  94?         links = repository.find_links_for_package(package)
16:05:17 Step #2 - "build":         95? 
16:05:17 Step #2 - "build":         96?         hashes = [f["hash"] for f in package.files]
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    6  ~/.poetry/lib/poetry/repositories/pypi_repository.py:245 in find_links_for_package
16:05:17 Step #2 - "build":        243? 
16:05:17 Step #2 - "build":        244?     def find_links_for_package(self, package):
16:05:17 Step #2 - "build":      ? 245?         json_data = self._get("pypi/{}/{}/json".format(package.name, package.version))
16:05:17 Step #2 - "build":        246?         if json_data is None:
16:05:17 Step #2 - "build":        247?             return []
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    5  ~/.poetry/lib/poetry/repositories/pypi_repository.py:319 in _get
16:05:17 Step #2 - "build":        317?     def _get(self, endpoint):  # type: (str) -> Union[dict, None]
16:05:17 Step #2 - "build":        318?         try:
16:05:17 Step #2 - "build":      ? 319?             json_response = self.session.get(self._base_url + endpoint)
16:05:17 Step #2 - "build":        320?         except requests.exceptions.TooManyRedirects:
16:05:17 Step #2 - "build":        321?             # Cache control redirect loop.
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    4  ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:543 in get
16:05:17 Step #2 - "build":        541? 
16:05:17 Step #2 - "build":        542?         kwargs.setdefault('allow_redirects', True)
16:05:17 Step #2 - "build":      ? 543?         return self.request('GET', url, **kwargs)
16:05:17 Step #2 - "build":        544? 
16:05:17 Step #2 - "build":        545?     def options(self, url, **kwargs):
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    3  ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:530 in request
16:05:17 Step #2 - "build":        528?         }
16:05:17 Step #2 - "build":        529?         send_kwargs.update(settings)
16:05:17 Step #2 - "build":      ? 530?         resp = self.send(prep, **send_kwargs)
16:05:17 Step #2 - "build":        531? 
16:05:17 Step #2 - "build":        532?         return resp
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    2  ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:643 in send
16:05:17 Step #2 - "build":        641? 
16:05:17 Step #2 - "build":        642?         # Send the request
16:05:17 Step #2 - "build":      ? 643?         r = adapter.send(request, **kwargs)
16:05:17 Step #2 - "build":        644? 
16:05:17 Step #2 - "build":        645?         # Total elapsed time of the request (approximately)
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    1  ~/.poetry/lib/poetry/_vendor/py3.7/cachecontrol/adapter.py:53 in send
16:05:17 Step #2 - "build":         51?             request.headers.update(self.controller.conditional_headers(request))
16:05:17 Step #2 - "build":         52? 
16:05:17 Step #2 - "build":      ?  53?         resp = super(CacheControlAdapter, self).send(request, **kw)
16:05:17 Step #2 - "build":         54? 
16:05:17 Step #2 - "build":         55?         return resp
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   ConnectionError
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   at ~/.poetry/lib/poetry/_vendor/py3.7/requests/adapters.py:498 in send
16:05:17 Step #2 - "build":       494?                     low_conn.close()
16:05:17 Step #2 - "build":       495?                     raise
16:05:17 Step #2 - "build":       496? 
16:05:17 Step #2 - "build":       497?         except (ProtocolError, socket.error) as err:
16:05:17 Step #2 - "build":     ? 498?             raise ConnectionError(err, request=request)
16:05:17 Step #2 - "build":       499? 
16:05:17 Step #2 - "build":       500?         except MaxRetryError as e:
16:05:17 Step #2 - "build":       501?             if isinstance(e.reason, ConnectTimeoutError):
16:05:17 Step #2 - "build":       502?                 # TODO: Remove this in 3.0.0: see #2811
16:05:17 Step #2 - "build":

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 9
Comments: 52 (12 by maintainers)

Commits related to this issue

Do not reuse requests sessions When running under certain environment, reuse of request sessions maybe causing network connectivity issues. This change ensures that request sessions are not reused ac... — committed to abn/poetry by abn 4 years ago
Do not reuse requests sessions When running under certain environment, reuse of request sessions maybe causing network connectivity issues. This change ensures that request sessions are not reused ac... — committed to python-poetry/poetry by abn 4 years ago

Most upvoted comments

Still experiencing this problem on 1.5.1

ConnectionResetError
 
   [Errno 104] Connection reset by peer
 
   at /usr/lib/python3.7/ssl.py:929 in read
        925│         if self._sslobj is None:
        926│             raise ValueError("Read on closed or unwrapped SSL socket.")
        927│         try:
        928│             if buffer is not None:
     →  929│                 return self._sslobj.read(len, buffer)
        930│             else:
        931│                 return self._sslobj.read(len)
        932│         except SSLError as x:
        933│             if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
 
 The following error occurred when trying to handle this error:
 
 
   ProtocolError
 
   ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
 
   at /opt/poetry/venv/lib/python3.7/site-packages/urllib3/response.py:461 in _error_catcher
       457│                 raise ReadTimeoutError(self._pool, None, "Read timed out.")
       458│ 
       459│             except (HTTPException, SocketError) as e:
       460│                 # This includes IncompleteRead.
     → 461│                 raise ProtocolError("Connection broken: %r" % e, e)
       462│ 
       463│             # If no exception is thrown, we should avoid cleaning up
       464│             # unnecessarily.
       465│             clean_exit = True
 
 The following error occurred when trying to handle this error:
 
 
   ChunkedEncodingError
 
   ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
 
   at /opt/poetry/venv/lib/python3.7/site-packages/requests/models.py:818 in generate
        814│             if hasattr(self.raw, "stream"):
        815│                 try:
        816│                     yield from self.raw.stream(chunk_size, decode_content=True)
        817│                 except ProtocolError as e:
     →  818│                     raise ChunkedEncodingError(e)
        819│                 except DecodeError as e:
        820│                     raise ContentDecodingError(e)
        821│                 except ReadTimeoutError as e:
        822│                     raise ConnectionError(e)

omerbe-nym on Jul 6, 2023

Still experiencing this problem on 1.2.2

illyakaynov on Nov 18, 2022

We’re experiencing this issue as well very consistently in our repositories that include a link to our private Azure package feed. We’ve tried the solution posted above to no avail. Note this is on poetry version 1.1.12. Error:

Updating dependencies
Resolving dependencies... (295.3s)

  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811

We’ve noticed that when we remove our private packages and package feed from our list, the problem goes away immediately; also dependency resolution is much much faster (in this case, from 100-300 seconds with failure, down to ~20 seconds with success).

rbudnar on Dec 6, 2021

We’re still seeing exactly this problem on our CI builds (azure pipelines) with poetry-1.1.4, our docker base image is python:3.6-slim-buster just like OP:

#18 13.66   • Installing black (19.10b0)
#18 13.83 
#18 13.83   ConnectionError
#18 13.83 
#18 13.83   ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
#18 13.83 
#18 13.83   at /usr/local/lib/python3.6/site-packages/requests/adapters.py:498 in send
#18 13.87       494│                     low_conn.close()
#18 13.87       495│                     raise
#18 13.87       496│ 
#18 13.87       497│         except (ProtocolError, socket.error) as err:
#18 13.87     → 498│             raise ConnectionError(err, request=request)
#18 13.87       499│ 
#18 13.87       500│         except MaxRetryError as e:
#18 13.87       501│             if isinstance(e.reason, ConnectTimeoutError):
#18 13.87       502│                 # TODO: Remove this in 3.0.0: see #2811

Since there’s a pretty big list of packages to install, this happens almost every build. We’re pinning to poetry < 1.1 for now as a workaround.

Korijn on Nov 5, 2020

This was resolved with 1.1.4.

abn on Oct 24, 2020

@sloev as I mentioned on the pull request (https://github.com/python-poetry/poetry/pull/3230#issuecomment-713551777), this is an open source community project, not a product you have a paid support for. This is maintained by a small group of volunteers who spend their spare time on this. You demanding issues be fixed and spamming issue threads is not going to make anything go any faster. While we can appreciate that this is causing issues with your workflow, we have mentioned in the release post how to disable the experimental installer to unblock yourself. Please keep the conversation constructive and civil.

abn on Oct 22, 2020

I confirm the same issue with poetry 1.6.1, the only thing that brings stability (in a turtle-speed way though) is poetry config installer.parallel false

we noticed however that this problem happens only to customers with nvidia-packages (cuda, cublas, cudnn, etc.) in the installation chain and only by building a docker image on Windows.

gehirndienst on Sep 26, 2023

Still experiencing this problem on 1.4.x poetry config installer.parallel false works almost always

poetry config installer.max-workers 4 still fails most of the time poetry config installer.max-workers 2 works

I think there should be a problem with the way parallel works.

mohghaderi on May 9, 2023

@omerbe-nym @ezorita

If you’re still experiencing this, it’s possible that your docker container is reporting a ton of cores (whether that is true or not, see below) when os.cpu_count() is called by poetry. See https://github.com/python-poetry/poetry/issues/3219#issuecomment-955496032

The max number of workers has no upper bound by default, meaning hundreds of connections might be attempted and this error occurs. Setting max-workers to a reasonable value, e.g. somewhere between 5 - 8, will likely resolve this.

POETRY_INSTALLER_MAX_WORKERS=5 poetry install

We encountered this on CircleCI - CircleCI’s Python convenience image reports 36 cores regardless of which resource class you’re using.

coltonb-arcadia on Aug 30, 2023

Hi folks, I’m also having what I think is the same/a related problem failing on my builds, poetry 1.1.6 , maybe 50% of calls to a private package repository (Azure Artifacts Feed) are failing with the below. Interested to know if anybody else has a workaround/ solved.

$  poetry install
$ make test-all
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.7.10/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.7.10/x64/lib
No suitable keyring backends were found
Using a plaintext file to store and retrieve credentials
Creating virtualenv ***_-py3.7 in /home/runner/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.7/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811
      ```

btjones-me on Apr 21, 2021

+1 – happening on poetry 1.1.13 in a docker container 😦

roscioli on Jun 1, 2022

This happened for me in CI with a build machine with 32 cores.

GitLab CI
Poetry 1.1.13
Docker executor running python:3.9

  ConnectionError
  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/tomli/2.0.1/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb8b979fdf0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
  at /usr/local/lib/python3.9/site-packages/requests/adapters.py:519 in send
      515│             if isinstance(e.reason, _SSLError):
      516│                 # This branch is for urllib3 v1.22 and later.
      517│                 raise SSLError(e, request=request)
      518│ 
    → 519│             raise ConnectionError(e, request=request)
      520│ 
      521│         except ClosedPoolError as e:
      522│             raise ConnectionError(e, request=request)
      523│

johnthagen on May 31, 2022

Hey guys!

I’ve encountered this issue on a machine with a lot of CPUs (128). Looks like poetry install is basically DDOSing PYPI if you have a lot of cores.

Temporal solution: poetry config installer.parallel false. Docker solution: limit the number of CPUs available to docker build. (--cpuset-cpus <num>).

Perhaps a poetry config installer.max_cpus setting would be a good final solution.

danielgafni on May 31, 2022

@DustinMoriarty yeah; we noticed that. This should go away once 1.1.4 is out. Hopefully the fix in #3230 is sufficient.

abn on Oct 22, 2020

Hmm, we saw this intermittently in our own CI a while ago, but was restricted to Cirrus CI FreeBSD instances (which runs on Google Cloud).

We might be able to do a bug fix to try recreate the session, but I suspect the issue might be at the process/scoket level.

@winpat what environment are you seeing your errors in?

I am really inclined to move http logic to httpx in the next minor to handle concurrent scenarios a bit better.

abn on Oct 16, 2020

hey @abn ! I’m sorry, I totally forgot about this thread.

I’ve found a PR that implements exactly this, it looks like it’s almost ready.

danielgafni on Nov 10, 2021

@danielgafni can you open a new issue for this? The workers need to be configurable. We definitely are a bit greedy, but then again I suspct what we need to connection pooling for repository sessions so installations can continue faster if all dependencies are already cached for example.

https://github.com/python-poetry/poetry/blob/c6b19b348d86390135e16ac17e426e5c41202b16/poetry/installation/executor.py#L75

abn on Oct 30, 2021

@SimonBiggs Is there a different issue that I should follow to track updates on this? The connection error issues are making it so that I have to disable the new installer for the time being.

taranlu-houzz on Mar 15, 2021

Since there’s a pretty big list of packages to install, this happens almost every build. We’re pinning to poetry < 1.1 for now as a workaround.

As per https://python-poetry.org/blog/announcing-poetry-1-1-0.html#brand-new-installer adding something like the following to your CI builds might work:

https://github.com/pymedphys/pymedphys/blob/765f6229105f221c3dd75ff9927e62dfde1b3972/.github/workflows/test-and-deploy.yml#L154-L156

SimonBiggs on Dec 22, 2020

~when is this gonna be fixed? this morning: again 9 consecutive travis builds before i got thru my poetry install step. poetry version: 1.1.2~

i’ve now recognized my posts here and on the other issues were unnecesarily blunt and had a tone i dont stand for. So i apologize for my behavior and wish no harm was done. I was under stress and this was an accidental side-effect. Hope you all have a nice day and i will see if i can find a way to contribute constructively to this great project going forward. …johannes

sloev on Oct 22, 2020

@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.

I’ve run it four times now and it’s worked every time. I believe the PR does fix this issue.

maroux on Oct 20, 2020

@abn Thanks for your help! I’ve added your fix to a couple of different projects. It will take a day or two until I can whether it’s working or not.

winpat on Oct 20, 2020