bullmq: [Bug]: Worker stops processing jobs

Version

^5.41.3

Platform

NodeJS

What happened?

Some workers will eventually stop processing jobs until the node process is restarted, at which point all jobs will start running again.

The worker list for the queue returns a connection, but its idle time and age are the same which makes me thing the worker is created but is stalled for some reason except no jobs are actually stalled when I check the queue status using bull-board.

Sorry if this isn’t the right forum to post this

How to reproduce.

I don’t really know how to reproduce this issue

Relevant log output

// return of queue.getWorkers
[
  {
    id: '1711546',
    addr: 'redacted',
    laddr: 'redacted',
    fd: '29',
    name: 'myjob',
    age: '11336',
    idle: '11336',
    flags: 'b',
    db: '0',
    sub: '0',
    psub: '0',
    ssub: '0',
    multi: '-1',
    qbuf: '0',
    'qbuf-free': '0',
    'argv-mem': '40',
    'multi-mem': '0',
    rbs: '1024',
    rbp: '0',
    obl: '0',
    oll: '0',
    omem: '0',
    'tot-mem': '2608',
    events: 'r',
    cmd: 'bzpopmin',
    user: 'default',
    redir: '-1',
    resp: '2',
    rawname: 'bull:SW1hcFNjYW5NYWlsYm94'
  }
]

Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 32 (15 by maintainers)

Most upvoted comments

So the facts are:

  • the queue has delayed jobs, seem like they are delayed like 5 minutes.
  • at least one worker is online
  • there is a disconnection event
  • jobs are not processed anymore

Based on this facts it would be very useful to have a reproducible test case. Also something to test is using Redis monitor command after the worker stops processing, to see what is going on in Redis, as normally you should see a BZPOPMIN command (and others too) every 5 seconds when the queue is idle.

Can you increase the shard numbers on stage to see if you can reproduce it?

Good day, @manast! We’ve been encountering a similar issue for about a week. The worker has stopped consuming jobs, and we don’t see any BZPOPMIN commands in the logs after that, despite other queues continuing to work. What’s interesting is that this is random behavior - any worker could stop at any time, not just that one. The worker itself is fairly simple—it just sends messages to Kafka. It’s unlikely that this is related to the async function, as our logs confirm that it completes successfully.

We’re running on an AWS ElastiCache Cluster (Valkey 8.0.1) with 2 shards. We’re running service with DEBUG:ioredis*, so I can share anything you need that could help resolve this issue.

Versions:

"@nestjs/bullmq": "^11.0.0",
"bullmq": "^5.53.0",
"ioredis": "^5.6.1",

What we did a week ago: Updated BullMQ from "^5.51.1" to "^5.52.1" Updated NodeJS from 23.11 to 24.1

We have the same behavior with Redis Sentinel in the k8s, using

"ioredis": "^5.6.0",
"bullmq": "^5.44.4",
"@nestjs/bullmq": "^10.2.3",

We have several processors and they just stop processing the jobs one by one at random time. We have checked that there are no Active jobs in the queue when it stuck.

In logs we periodically observe Error: Connection is closed. from Redis, and All sentinels are unreachable errors, but they do not always lead to stuck jobs. Also, our other services, that are using ioredis are working fine and not experiencing any issues with Redis.

Appreciate it if you could advise how we could debug such case.