daphne: Django Channels memory leaks on opening new connections

Hello! First of all, this is a great project and I wanted to thank all the maintainers for keeping it amazing!

For last two months, django-channels websockets on our production service have been periodically failing with OOM. Having researched it, it looks like a django-channels memory leak.

I think it may be related to opening new channel connections and then improperly closing them in channels_layer object. Opening multiple browser tabs leading to a single channels group increases memory usage, and closing them does not release the memory (the disconnect occurs though).

I have also managed to reproduce it on a simple project with minimal dependencies and steps to reproduce, so please feel free to check it out: https://github.com/yuriymironov96/django-channels-leak. This is a project based on django-channels tutorial.

Here are dependencies of the sample project:

python = “^3.8”
channels = “^3.0.3”
channels-redis = “^3.2.0”
channels-rabbitmq = “^3.0.0”
Pympler = “^0.9”
daphne = “^3.0.2”

I have tries both channels_redis and channels_rabbitmq, multiple servers (django debug server, daphne, uvicorn) and the issue still persists.

The sample benchmarks are:

Server is launched, no browser tabs open: 49.3 MB;
First browser tab opened: 51.2 MB;
20 browser tabs opened: 52.2 MB;
All browser tabs closed: 52.2 MB;
20 browser tabs opened: 52.6 MB;
All browser tabs closed: 52.6 MB;

It may look like a minor leak at this rate, but it is scales quickly and occurs frequently due to high load of our application.

Could you please have a look at it and share any ideas you have?

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 11
Comments: 26 (3 by maintainers)

Most upvoted comments

After a week of usage of Uvicorn instead Daphne in production environment, I’m now fully convinced that this leak is related to Daphne, not channels. In my use case, results are significant:

memoryusagedaphne-png-2362×664-

Not only that, the switch to Uvicorn proved to have significant effect on CPU usage as well. Its lower than that of Daphne. As a result, we have not only addressed memory leak, but also improved website performance as well.

revoteon on Jul 1, 2021

Thanks for looking into this @pgrzesik. We are using Python 3.8 and don’t use the ddtrace library.

aaditya-ridecell on Aug 15, 2023

@BSVogler Thanks a lot for the update 🙇

@carltongibson I’m currently trying to nail down the issue that we’re encountering. If I manage to pin it down to specific behavior of daphne I’ll make sure to share a minimal reproducible case.

pgrzesik on Aug 3, 2023

We ended up here as we were looking for the source of memory leaks. We are currently using daphne on python3.11 docker x86 image. Daphne on arm seems to be fine though.

BSVogler on Mar 29, 2023

Thanks for the effort @pgrzesik – even a partial result helps narrow things down. 🎁

carltongibson on Aug 15, 2023

Hey 👋 After digging deeper into it, it seems like the issue on our end is related to some weird interaction between daphne and ddtrace library on Python 3.11 (Python 3.10 is fine, we have it running for a long time with the same setup). Without ddtrace we don’t experience these segfaults, similarily with ddtrace but without daphne (e.g. on our other apps), we also don’t experience these issues. Given the above, I’m afraid I won’t be able to help with reproduction as it seems to be more complex than initially thought.

@aaditya-ridecell What Python version are you using?

pgrzesik on Aug 15, 2023

@pgrzesik Thank you for taking this up! We really cannot find a root cause and might end up removing channels code.

aaditya-ridecell on Aug 4, 2023