bullmq: [Bug]: Worker stuck, not pulling jobs
Version
v5.3.3
Platform
NodeJS
What happened?
Periodically, one of our Workers stops pulling jobs and requires a full restart of the node process to get going again.
Symptom notes:
- stalled timer check is still running (confirming that Event Loop, Task Queue and MicroTask Queue aren’t stuck in infinite loops, since the check uses both setTimeout and async/await, and console.logs I have added to display when the timer is refreshed is showing in our node logs).
- No ‘error’ events being emitted on the worker
- No ‘closing’ event being emitted on the worker
- There is generally a job stuck as Active in the Queue, but the job had been stuck for a long time (with other jobs still running on the worker after it got stuck) before the Worker stopped pulling jobs. It is not always the same job type/data that I see stuck (this note added before I managed to find a reproducible scenario)
- Worker was performing the
asyncFifoQueue.fetch();when it stopped running - Other Workers using other Queues are not affected
- no
bzpopmincalls for the affected Queue, but those calls for other Queues - removing the
limiterworker option stops the issue occuring. Changing the values of this option also changes at what point it will crash
Using Sandboxed workers and using concurrency (5 for the latter, IIRC).
I also saw ready events after the failure on that Worker every 10 minutes until I restarted it (note: our Redis idle timeout is 10 minutes).
Current theory:
- Active job gets stuck (infinite loop/pending promise/etc)
- Lots of jobs come through into the queue
- Worker hits the set up job limit
- Code stops running the while loop (https://github.com/taskforcesh/bullmq/blob/46f4bcc77b8f458c8c31b45351ce4e7d60f2c96c/src/classes/worker.ts#L492) and goes into the second loop for pulling jobs (https://github.com/taskforcesh/bullmq/blob/46f4bcc77b8f458c8c31b45351ce4e7d60f2c96c/src/classes/worker.ts#L538)
- fetch awaits on the the pending
processJobpromise - Worker is stuck and can’t go back to the start of the loop to pull any more jobs to process them
How to reproduce.
Able to reproduce! Have two functions:
async function test() {
try {
return 'Success'
}
catch (err) {
throw err;
}
}
async function delaytest() {
try {
// while(true) {
// }
return await new Promise((resolve, reject) => {});
}
catch (err) {
throw err;
}
}
Create a worker with a job limit:
const theWorker= new Worker(
thequeuename,
thesandboxurl,
{ connection: newRedisConnection(), concurrency: 5, limiter: {
max: 50,
duration: 5*1000
} }
);
Add one of the delaytest jobs, then 1000 of the test ones. It runs about 40-50 before it dies.
Relevant log output
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 19 (16 by maintainers)
Commits related to this issue
- fix(worker): avoid dangling jobs to hang the queue with rate limit fixes #3289 — committed to taskforcesh/bullmq by manast a year ago
- fix(worker): avoid dangling jobs to hang the queue with rate limit fixes #3289 (#3297) — committed to nagyist/bullmq by manast a year ago
I have now a minimal test that reproduces the issue, it can be reproduced just with 3 jobs, where 1 of them never completes. I am investigating the root cause.