requests: Connection reset by peer when sending POST

Client code that uses Requests module to send data via HTTP POST encounters a ConnectionResetError. The entire operation (composed of multiple POST requests to a small set of service endpoints) can sometimes succeed, but most of the time, it fails with this error.

Expected Result

Operation succeeds (or fails) without connection issues.

Actual Result

The operation fails with ConnectionResetError.

Additional Information

It’s a little difficult to provide basic reproduction for the issue as we’re running into this problem with a test payload that’s specific to our system. The server (peer) is a Java application that’s configured to terminate/reset connection after a given time of no use (idle). The client sends multiple one-off POST requests, but it seems like internally, the connections are being reused similar to issue #4506, and the operation eventually runs into a connection that’s already been reset and raises the error. However, unlike #4506, we are not using Sessions.

Here are some tracebacks that could hopefully describe the problem better:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

...

Traceback (most recent call last):
  ...
  File "../client-code.py", line 322, in createFile
    r = requests.post(fileSubmissionsUrl, data=json.dumps(fileToCreateObject), headers=self.headers)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.3.1"
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.1"
  },
  "platform": {
    "release": "4.14.42-61.37.amzn2.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1000211f",
    "version": "18.0.0"
  },
  "requests": {
    "version": "2.20.1"
  },
  "system_ssl": {
    "version": "20000000"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": true
}

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 26 (1 by maintainers)

Most upvoted comments

Hello… The Persistent Connectionts features are not handling the client side when it requested a Connection Keep Alive, because at lease for Linux based OS the TCP Keep Alive Feature is not enabled, or when Enabled it is set to 2 start working after 2 hours, so I found that the problem iis because the OS is not handling the flo control correctly you can fix it in some ways…

For Python in Linux

import requests import socket from urllib3.connection import HTTPConnection HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [ (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) #Enables the feature ,(socket.SOL_TCP, socket.TCP_KEEPIDLE, 45) #Overrides the time when the stack willl start sending KeppAlives after no data received on a Persistent Connection ,(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10) #Defines how often thoe KA will be sent between them ,(socket.SOL_TCP, socket.TCP_KEEPCNT, 6) #How many attemps will your code try if the server goes down before droping the connection.

now you can Instanciate your Sessions or Requests…

Without TCP Keep Alive

With TCP Keep Alive

If the client reach the unhandled session timeout by the server…now the server will close the connection , releasing the connection back to the pool insteadd of dropoing it…

There are some other ways to handle it… depending on your OS and in your needs… I hope it solve some of your issues

jakermx on Feb 4, 2021

Any solutions?

hedza06 on Jan 14, 2021

A workaround that did work was to simply catch the error and retry. I am still not sure whether the client or the server is dropping the connection, and why they are doing so, but hopefully this workaround helps somebody else.

Note that both the request creation (requests.post) and the response read/parse (response.json()) are within the try block.

        try:
            response = requests.post(self.DATASTORE_URL+"/datastreams/find_entries/timestamp", json=post_msg)
            print("response = %s" % response)
            response.raise_for_status()
            ret_list = response.json()["phone_data"]
        except Exception as e:
            # Hacky copy-paste of original code, TODO refactor into separate function
            print("Got %s error %s, retrying" % (type(e).__name__, e))
            time.sleep(10)
            response = requests.post(self.DATASTORE_URL+"/datastreams/find_entries/timestamp", json=post_msg)
            print("response = %s" % response)
            response.raise_for_status()
            ret_list = response.json()["phone_data"]

shankari on Aug 21, 2019