locust: OOM error with master/slaves setup (zeromq, windows)
Hi !
Describe the bug
An out of memory error occurs with ZeroMQ trying to allocate a crazy amount of memory in decoded_allocator, sometime up to several petabytes. This might very well be a ZeroMQ bug :
OUT OF MEMORY (bundled\zeromq\src\decoder_allocators.cpp:89)
I added some logs and recompiled pyzmq to check what’s going on. Upon further investigation, _max_counters seems to take a crazy value at some point. See zmq_logs.txt As you can see, allocator instance 0x0000016A9270F700 is constructed with _max_counters=249, but before crash its value has changed to 1557249601288, which causes a malloc of several terabytes.
Steps to reproduce
Sorry, I couldn’t find a surefire way to reproduce this one. It seems kind of random. It sometime happens before the test is even started, sometime when the test is stopped. Sometime it doesn’t happen at all. It does seem to happen more often when stopping a test in the web UI. Simply run the ps1 attached and do some stuff in the web UI.
Environment
- OS: Windows 10.0.18362.778
- Python version: 3.6
- Locust version: 0.14.6
- Locust files : test_locust.zip
I managed to repro the bug on two computers : my work computer and my personal computer. Both are on Windows 10/Python 3.6 that comes with VS2017, but my personal computer has a pristine python environent, just ran pip install locustio.
Am I doing something I’m not supposed to ?
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 30 (3 by maintainers)
Commits related to this issue
- Bump dependency on pyzmq to fix #1372 (OOM on windows) — committed to locustio/locust by cyberw 3 years ago
- Merge pull request #1839 from locustio/bump-dependency-on-pyzmq-to-fix-windows-OOM-issue Bump dependency on pyzmq to fix #1372 (OOM on windows) — committed to locustio/locust by cyberw 3 years ago
@RichardLions Good job updating the other bug (zeromq/pyzmq#1555) and finding a possible cause. Silly me, I thought the error (out of memory) could be something to do with running out of memory. 😃 I did try to run some python memory profilers but all I saw was a very flat memory allocation over time and nothing alarming.
pyzmq 22.2.1 has been released containing the fix for this issue.
https://github.com/zeromq/pyzmq/releases/tag/v22.2.1
@RichardLions has a pull request that fixes this in the pyzmq project. I implemented it manually and tested and it works. See zeromq/pyzmq#1555
Pull Request: zeromq/pyzmq#1560
I also have this issue.
FATAL ERROR: OUT OF MEMORY (C:\projects\libzmq\src\decoder_allocators.cpp:85)
I get it a majority if the time when just creating my master and worker nodes. I’d say 3 out of 4 attempts fail. If that part passes, sometimes it fails with the same error after I start my load test.
versions: python 3.9.5 locust 1.5.3 pyzmq 22.1.0 OS: Windows 10, Windows Server 2016 Free Memory at time of failure: 7.3 GB of 16 GB on my Windows 10 box
I will check out pyzmq, but I wanted to post here for the sake of visibility (i.e., it isn’t just a few people getting this error, when I talked with the guy who recommended Locust, he said “Oh yeah it does that all the time. I just keep trying until it works.” Personally I’d rather fix it. So I’ll see if the folks at pyzmq have this on their radar already.
Thanks.
Hi! Sorry, I have nothing to add here. You probably already know a lot more than me 😃
It’s fine, our actual locusts are run on Linux anyway. I’ll simply keep working on WSL locally and try to find a fix if I have some time.
Sure thing ! I’ll try on WSL, if that works for you. Worst case, I can set up a Linux VM.