httpx: Memory leak when creating lots of AsyncClient contexts

Checklist

  • The bug is reproducible against the latest release and/or master.
  • There are no similar issues or pull requests to fix it yet.

Describe the bug

After creating an AsyncClient context (with async with), it does not seem to be garbage collected, that can be a problem for very long running services that might create a bunch of them and eventually run out of memory

To reproduce

import httpx
import gc
import asyncio

print(f"httpx version: {httpx.__version__}")


async def make_async_client():
    async with httpx.AsyncClient() as client:
        await asyncio.sleep(10)


async def main(n):
    tasks = []
    for _ in range(n):
        tasks.append(make_async_client())
    print(f"Creating {n} contexts, sleeping 10 secs")
    await asyncio.wait(tasks)


asyncio.run(main(2000))
print("Finished run, still using lots of memory")
gc.collect()
input("gc.collect() does not help :(")

Comparison with aiohttp

import aiohttp
import asyncio

print(f"aiohttp version {aiohttp.__version__}")


async def make_async_client():
    async with aiohttp.ClientSession() as client:
        await asyncio.sleep(10)


async def main(n):
    tasks = []
    for _ in range(n):
        tasks.append(make_async_client())
    print(f"Creating {n} contexts and sleeping")
    await asyncio.wait(tasks)


asyncio.run(main(200000))
input("Finished run, all memory is freed")

Expected behavior

Memory gets freed, after exiting the async context, like for aiohttp

Actual behavior

Memory does not get freed, even after explicitly calling gc.collect()

Debugging material

Environment

  • OS: Linux (many versions)
  • Python version: 3.8.3
  • HTTPX version: both 0.12.1 and master
  • Async environment: both asyncio and trio
  • HTTP proxy: no
  • Custom certificates: no

Additional context

I understand typically you need to have only one async ClientSession, but it shouldn’t leak memory anyway, for very long running processes it can be a problem

Thanks for this great library! If you’re interested I can try to debug this issue and send a PR

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 29 (9 by maintainers)

Commits related to this issue

Most upvoted comments

You don’t need to create 2000 clients - create a single client, and reuse it throughout. ⬆️

import httpx
import gc
import asyncio

print(f"httpx version: {httpx.__version__}")


async def make_async_client():
    async with httpx.AsyncClient(verify=False) as client:
        await client.request(method='get', url='https://gorest.co.in/public/v1/users')
        await asyncio.sleep(10)


async def main(n):
    tasks = []
    for _ in range(n):
        tasks.append(make_async_client())
    print(f"Creating {n} contexts, sleeping 10 secs")
    await asyncio.wait(tasks)


asyncio.run(main(2000))
print("Finished run, still using lots of memory")
gc.collect()
input("gc.collect() does not help :(")

memory leak again. What am I doing wrong?

httpx: 0.18.2 python: 3.7.9 docker: 20.10.7 ubuntu: 20.04

I can’t exactly replicate that, no. It’ll will use 1.5GB memory while it has 2000 instances simultaneously in memory, although it’ll free up once they’re out of scope.

One thing we might want to consider in any case is globally caching our SSL contexts. They’re a little bit slow to create, and they’re memory hungry, so it’d probably make sense for us to cache a small number of them so that users who’re (unnecessarily) creating lots of clients aren’t being negatively impacted.