bullmq: [Bug]: Worker stops processing jobs
Version
^5.41.3
Platform
NodeJS
What happened?
Some workers will eventually stop processing jobs until the node process is restarted, at which point all jobs will start running again.
The worker list for the queue returns a connection, but its idle time and age are the same which makes me thing the worker is created but is stalled for some reason except no jobs are actually stalled when I check the queue status using bull-board.
Sorry if this isn’t the right forum to post this
How to reproduce.
I don’t really know how to reproduce this issue
Relevant log output
// return of queue.getWorkers
[
{
id: '1711546',
addr: 'redacted',
laddr: 'redacted',
fd: '29',
name: 'myjob',
age: '11336',
idle: '11336',
flags: 'b',
db: '0',
sub: '0',
psub: '0',
ssub: '0',
multi: '-1',
qbuf: '0',
'qbuf-free': '0',
'argv-mem': '40',
'multi-mem': '0',
rbs: '1024',
rbp: '0',
obl: '0',
oll: '0',
omem: '0',
'tot-mem': '2608',
events: 'r',
cmd: 'bzpopmin',
user: 'default',
redir: '-1',
resp: '2',
rawname: 'bull:SW1hcFNjYW5NYWlsYm94'
}
]
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 32 (15 by maintainers)
So the facts are:
Based on this facts it would be very useful to have a reproducible test case. Also something to test is using Redis monitor command after the worker stops processing, to see what is going on in Redis, as normally you should see a BZPOPMIN command (and others too) every 5 seconds when the queue is idle.
Can you increase the shard numbers on stage to see if you can reproduce it?
Good day, @manast! We’ve been encountering a similar issue for about a week. The worker has stopped consuming jobs, and we don’t see any
BZPOPMINcommands in the logs after that, despite other queues continuing to work. What’s interesting is that this is random behavior - any worker could stop at any time, not just that one. The worker itself is fairly simple—it just sends messages to Kafka. It’s unlikely that this is related to the async function, as our logs confirm that it completes successfully.We’re running on an AWS ElastiCache Cluster (Valkey 8.0.1) with 2 shards. We’re running service with
DEBUG:ioredis*, so I can share anything you need that could help resolve this issue.Versions:
What we did a week ago: Updated BullMQ from
"^5.51.1"to"^5.52.1"Updated NodeJS from23.11to24.1We have the same behavior with Redis Sentinel in the k8s, using
We have several processors and they just stop processing the jobs one by one at random time. We have checked that there are no Active jobs in the queue when it stuck.
In logs we periodically observe
Error: Connection is closed.from Redis, andAll sentinels are unreachableerrors, but they do not always lead to stuck jobs. Also, our other services, that are using ioredis are working fine and not experiencing any issues with Redis.Appreciate it if you could advise how we could debug such case.