requests: [Tracking] slow leak in latest requests with https connections

This seems unrelated to https://github.com/kennethreitz/requests/issues/1685 as this is specific to https connections and I’m specifically closing the session and response connections.

I’ve spent a few days investigating a memory leak we’ve seen with the requests module and narrowed it down to three issues:

1. when using pyOpenSSL with requests 2.13.0 in python 3.5.2 there’s one leak:
   1. with pyOpenSSL with a local server there’s a leak, removing pyOpenSSL resolves leak
2. when using pyOpenSSL with requests 2.13.0 in python 3.5.3 there’s two leaks:
   1. same as 1.1
   2. when connecting to servers on different hosts there’s a leak
3. in both 3.5.2 and 3.5.3 with external hosts the script is periodically unable to contact the host due to “network is unreachable” errors, presumably because some resource has been temporarily exhausted…eventually recovering and repeating the error.

I’ve investigated this some and it doesn’t seem to be a leak python objects (see tracemalloc util in script), or a leak sockets/file descriptors, so I believe it’s a native leak in either python or openssl given the behavior changed between 3.5.2. and 3.5.3. In our 3.5.2 tests we were using libssl1.0.1 and 3.5.3 libssl1.1.0 and upgrading libssl did not fix the issue.

I’ve created a test file here: https://gist.github.com/thehesiod/5dbd7f2bffbe0b850980e865f5649338

and a docker cloud image here: amohr/testing:stretch_request_leak based on:

FROM debian:stretch

COPY request_https_leak.py /tmp/request_https_leak.py

RUN apt-get update && \
    apt-get install -y python3.5 python3-pip git

RUN python3 -m pip install requests git+git://github.com/thehesiod/pyca.git@fix-py3#egg=calib setproctitle requests psutil

it constantly updates three files in the directory it’s run from:

• mprofile_*.dat: memory usage by number of requests
• nprofile_*.dat: number of net connections by number of requests
• fprofile_*.dat: number of file descriptors by number of requests

you can plot the memory usage with the python module: memory_profiler with mprof plot [FILENAME]

Here’s an example run from the image above running for 15,000 requests…while it seems to stabilize I’ve noticed that running it for a long time it continues with a fairly constant long term line…I think the python malloc just starts requesting larger chunks over time.

btw x-axis in the plots generated is request num, not time.

Also in the script is the ability to switch to a local server (which is one of the issues), when it generates the temp certs, hit enter for everything except common-name, and just put the name it tells you to put there.

Next steps I think are attempting to try a native python ssl socket to see if the problem reproduces there to see if this is a native ssl issue (would be good to test aiohttp as well to see if it doesn’t reproduce if they have a work-around, and if it does they should know), and then a full valgrind investigation to see what’s leaking the memory.