httpx: Memory leak when creating lots of AsyncClient contexts
Checklist
- The bug is reproducible against the latest release and/or
master. - There are no similar issues or pull requests to fix it yet.
Describe the bug
After creating an AsyncClient context (with async with), it does not seem to be garbage collected, that can be a problem for very long running services that might create a bunch of them and eventually run out of memory
To reproduce
import httpx
import gc
import asyncio
print(f"httpx version: {httpx.__version__}")
async def make_async_client():
async with httpx.AsyncClient() as client:
await asyncio.sleep(10)
async def main(n):
tasks = []
for _ in range(n):
tasks.append(make_async_client())
print(f"Creating {n} contexts, sleeping 10 secs")
await asyncio.wait(tasks)
asyncio.run(main(2000))
print("Finished run, still using lots of memory")
gc.collect()
input("gc.collect() does not help :(")
Comparison with aiohttp
import aiohttp
import asyncio
print(f"aiohttp version {aiohttp.__version__}")
async def make_async_client():
async with aiohttp.ClientSession() as client:
await asyncio.sleep(10)
async def main(n):
tasks = []
for _ in range(n):
tasks.append(make_async_client())
print(f"Creating {n} contexts and sleeping")
await asyncio.wait(tasks)
asyncio.run(main(200000))
input("Finished run, all memory is freed")
Expected behavior
Memory gets freed, after exiting the async context, like for aiohttp
Actual behavior
Memory does not get freed, even after explicitly calling gc.collect()
Debugging material
Environment
- OS: Linux (many versions)
- Python version: 3.8.3
- HTTPX version: both 0.12.1 and
master - Async environment: both
asyncioandtrio - HTTP proxy: no
- Custom certificates: no
Additional context
I understand typically you need to have only one async ClientSession, but it shouldn’t leak memory anyway, for very long running processes it can be a problem
Thanks for this great library! If you’re interested I can try to debug this issue and send a PR
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 29 (9 by maintainers)
Links to this issue
Commits related to this issue
- [mod] searx.network.client: the same configuration reuses the same ssl.SSLContext before there was one ssl.SSLContext per client. see https://github.com/encode/httpx/issues/978 — committed to searxng/searxng by dalf 3 years ago
- [mod] searx.network.client: the same configuration reuses the same ssl.SSLContext before there was one ssl.SSLContext per client. see https://github.com/encode/httpx/issues/978 — committed to searxng/searxng by dalf 3 years ago
- [mod] searx.network.client: the same configuration reuses the same ssl.SSLContext before there was one ssl.SSLContext per client. see https://github.com/encode/httpx/issues/978 — committed to searx/searx by dalf 3 years ago
memory leak again. What am I doing wrong?
httpx: 0.18.2 python: 3.7.9 docker: 20.10.7 ubuntu: 20.04
I can’t exactly replicate that, no. It’ll will use 1.5GB memory while it has 2000 instances simultaneously in memory, although it’ll free up once they’re out of scope.
One thing we might want to consider in any case is globally caching our SSL contexts. They’re a little bit slow to create, and they’re memory hungry, so it’d probably make sense for us to cache a small number of them so that users who’re (unnecessarily) creating lots of clients aren’t being negatively impacted.