bullmq: [Bug]: Worker stopped processing jobs, and mostly delayed Jobs
Version
v5.4.2
Platform
NodeJS
What happened?
We have a service, where a worker runs, and processes jobs. After the processing is done, it will create another job, which is delayed (around 64 minutes). Today, I noticed that the service and worker stopped processing jobs. There were no error messages in the logs. When I used BullBoard (I use it as a UI to see jobs), I saw the jobs were still in the delayed state, and like 24 hours overdue.
When I restarted the service, and the worker started, it immediately started processing those delayed jobs. This is not the first it happened. Today I though first checked the delayed jobs.
In today’s incident, the service has been running for 4 days.
We run in EKS on AWS (NodeJS service, using Typescript). I use BullMQ Pro. And we are using Groups and each Group has a concurrency set to 1.
How to reproduce.
I don’t have any test code for this
Relevant log output
No Logs or error logs were produced
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 119 (45 by maintainers)
Commits related to this issue
- chore(release): 5.7.1 [skip ci] ## [5.7.1](https://github.com/taskforcesh/bullmq/compare/v5.7.0...v5.7.1) (2024-04-10) ### Bug Fixes * **worker:** use 0.002 as minimum timeout for redis version low... — committed to taskforcesh/bullmq by semantic-release-bot 3 years ago
- fix(worker): use 0.002 as minimum timeout for redis version lower than 7.0.8 (#2515) fixes #2466 — committed to taskforcesh/bullmq by roggervalf 3 years ago
- perf(worker): do not call bzpopmin when blockDelay is lower or equal 0 (#2544) ref #2466 — committed to taskforcesh/bullmq by roggervalf 3 years ago
- chore(release): 5.7.6 [skip ci] ## [5.7.6](https://github.com/taskforcesh/bullmq/compare/v5.7.5...v5.7.6) (2024-04-27) ### Bug Fixes * **redis-connection:** increase redis retry strategy backoff ([... — committed to taskforcesh/bullmq by semantic-release-bot 3 years ago
Since it seems that the original authors cannot reproduce the issue anymore, I will close it now, so that other users do not get lured into this one if they are experiencing a similar issue, but not this exact one, as that will just confuse everybody.
@tavindev you are welcome to open a new issue with the particular details for your use case.
It is not enough adding the line with the disconnect, you must remove the other 2 as well.
@lukas-becker0 seems like I am able to reproduce it following your instructions. I will keep you updated…
Hi, @manast I did some tests with the code you provided.
If I force close the redis connections with e.g.
redis-cli CLIENT KILL TYPE normal && redis-cli CLIENT KILL TYPE slave && redis-cli CLIENT KILL TYPE pubsubthe worker reconnects and continues to process jobs as expected.If however I kill/shutdown the entire redis server
docker-compose down, then wait a few seconds (~5 seconds) and then start redis again (docker-compose up -d), the worker reconnects but does not continue to process jobs (althoughisRunningreturns true) from the queue. The queue does then no longer see any connected worker.(Simulating a server restart and/or crash)
Is that the expected behaviour ?
docker-compose.yaml(Setting
redis-cli config set maxmemory-policy noevictionafter the first start ofcourse)I modified your code slightly:
I am reopening as it seems this is still not working, until we can reproduce it.
We rolled back to
5.4.0two weeks ago and have not experienced this issue since. Still curious as to the resolution here so we know when it may be safe to attempt to keep BullMQ up-to-date again. Thanks for all your support thus far.5.7.6 fixes it for us
Unfortunately we are still experiencing the issue with the latest version. Could you please let me know which version introduced this issue so we can downgrade to that specific version.