bullmq: [Bug]: Worker stuck, not pulling jobs

Version

v5.3.3

Platform

NodeJS

What happened?

Periodically, one of our Workers stops pulling jobs and requires a full restart of the node process to get going again.

Symptom notes:

  • stalled timer check is still running (confirming that Event Loop, Task Queue and MicroTask Queue aren’t stuck in infinite loops, since the check uses both setTimeout and async/await, and console.logs I have added to display when the timer is refreshed is showing in our node logs).
  • No ‘error’ events being emitted on the worker
  • No ‘closing’ event being emitted on the worker
  • There is generally a job stuck as Active in the Queue, but the job had been stuck for a long time (with other jobs still running on the worker after it got stuck) before the Worker stopped pulling jobs. It is not always the same job type/data that I see stuck (this note added before I managed to find a reproducible scenario)
  • Worker was performing the asyncFifoQueue.fetch(); when it stopped running
  • Other Workers using other Queues are not affected
  • no bzpopmin calls for the affected Queue, but those calls for other Queues
  • removing the limiter worker option stops the issue occuring. Changing the values of this option also changes at what point it will crash

Using Sandboxed workers and using concurrency (5 for the latter, IIRC). I also saw ready events after the failure on that Worker every 10 minutes until I restarted it (note: our Redis idle timeout is 10 minutes).

Current theory:

  1. Active job gets stuck (infinite loop/pending promise/etc)
  2. Lots of jobs come through into the queue
  3. Worker hits the set up job limit
  4. Code stops running the while loop (https://github.com/taskforcesh/bullmq/blob/46f4bcc77b8f458c8c31b45351ce4e7d60f2c96c/src/classes/worker.ts#L492) and goes into the second loop for pulling jobs (https://github.com/taskforcesh/bullmq/blob/46f4bcc77b8f458c8c31b45351ce4e7d60f2c96c/src/classes/worker.ts#L538)
  5. fetch awaits on the the pending processJob promise
  6. Worker is stuck and can’t go back to the start of the loop to pull any more jobs to process them

How to reproduce.

Able to reproduce! Have two functions:

async function test() {
   try {
      return 'Success'
   }
   catch (err) {
      throw err;
   }
}

async function delaytest() {
   try {
      // while(true) {

      // }
      return await new Promise((resolve, reject) => {});
   }
   catch (err) {
      throw err;
   }
}

Create a worker with a job limit:

const theWorker= new Worker(
    thequeuename,
    thesandboxurl,
    { connection: newRedisConnection(), concurrency: 5, limiter: {
      max: 50,
      duration: 5*1000
    } }
  );

Add one of the delaytest jobs, then 1000 of the test ones. It runs about 40-50 before it dies.

Relevant log output


Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 19 (16 by maintainers)

Commits related to this issue

Most upvoted comments

I have now a minimal test that reproduces the issue, it can be reproduced just with 3 jobs, where 1 of them never completes. I am investigating the root cause.