requests: requests.Session doesn't properly handle closed keep-alive sessions

Hi,

I think i found a solution and created pull requests for urllib3 -> urllib3/urllib3#1911 After it will be merged ( I hope ) , requests should raise MaxRetries error as it should , because they are setting max_retries = 0 to retry object of urllib3 by default - > this should be discussed if better not to leave default urllib3 value which is 3 , or pass 1.

More info -> https://bugs.python.org/issue3566 , https://hg.python.org/cpython/rev/eba80326ba53 , https://bugs.python.org/issue41345

I agree when you said that the Session is not handling as should the connections, and I think that the reasson of the adapters…the RFC for “Persistent” Connections is very weird…it defines that the client should start sending tcp keep alive packets after 2 hours of no data received… for a Service Oriented Server…it is not costable having idle sessions. I.E. A video Streming server…chaturbate for say something… or netflix, users navigate throu the options and then picjk a video or room, but too many times , we a, as userrs let the video playing and fall asleep…so…we are consuming resourses that at the end we are causining our provider to spend more so they will increaase thier prices and so on…

So in the case, you software raise an exception instead of handling as should the connection…it is because the server side, maybe no the server software it, the load balancers, sanboxes or any other box in tyhe middle.

Thats Why I SUGGESTED to enable a lower layer connection guard that wull ensure that the session…well the last request, keep open until the server says…WTF and clloses thw connection properly instead of just dropping it.

so…it is work of no just Requests team…it really depens on 7 teams…one per OSI layer, actually 4, heheh.

SoI am just tryiong to let you know that we can be creatives and find solutinos, and try to understnad that is the Module is called Request , they handle requests…and suport itself on other specific modules…

XD

@keuko short of writing a custom HTTPAdapter (overriding the send method) – I don’t know.

I use code such as the following and attach the adapter using Session.mount (see https://requests.readthedocs.io/en/master/user/advanced/#transport-adapters).

class CustomHTTPAdapter(HTTPAdapter):
    def send(self, *args, **kwargs):
        for attempt in range(ATTEMPTS):
            if attempt > 0:
                LOGGER.info("Retrying after connection error")
            try:
                return super().send(*args, **kwargs)
            except ConnectionError as exc:
                LOGGER.info("Connection error: %s", exc)
                saved = exc
                continue

        raise saved

@malthe Well, so, is this issue “somehow” resolveable ? In openstack where tons of requests are send to API, from time to time there is an issue related to this …

I’m not saying it’s regular bug, but how can be this fixed , is it possible ?

Even if this issue (for example) in openstack is in small amount, from time to time something just fail …

For what it’s worth, I don’t think there’s any workaround to this besides catching the ConnectionError explicitly - urllib3.Retry has never retried on a ConnectionError (as far I can tell) cf https://github.com/urllib3/urllib3/blob/master/src/urllib3/util/retry.py#L9 cc @eriol

Hello @sigmavirus24, many thanks for the fast reply and sorry if I did not manage to reply earlier. Since version 2.18.1-1 I dropped all the patches for requests, but I still have some patches on urllib3:

https://sources.debian.org/patches/python-urllib3/1.22-1/

the biggest one is to not use the vendored six module.