bullmq: [Bug]: Job goes to delayed status upon creation
Version
v5.40.0
Platform
NodeJS
What happened?
We have been experiencing some latency for our worker. It appears that it needs time to warm up, but it is unclear why this is happening. I thought that it was related to this fix that you guys had made, but the issue persists after upgrading from 5.35.0 to 5.40.0
I am using bullmq v5.40.0
Here is a screenshot of the two jobs being moved to delayed status, even though their delay is set to 0 (sometimes a negative number)
I have tried the following:
- Upgrade to latest version of
bullmq - Manually promote the job, when I do this the job moves to
waitingstatus for the same amount of time as it would have stayed in thedelayedstatus
According to the docs here, if there is an error and backoff strategy is when adding the job then it goes to delayed status, but how can I visualize that to diagnose what the root cause is?
How to reproduce.
Here is the code for when we instantiate our worker. The worker lives in an Azure App Service
// Start the worker
this.worker = new Worker(HIP_BULL_QUEUE, this.process, {
name: 'hippo-worker',
concurrency: 5,
connection: {
host: HIP_BULL_REDIS_HOST,
port: HIP_BULL_REDIS_PORT,
password: HIP_BULL_REDIS_PASS,
db: HIP_BULL_REDIS_DB
},
metrics: {
maxDataPoints: MetricsTime.TWO_WEEKS
}
});
Here is the code to add the job
const job = await Queue.add({
category: JOB_CATEGORY,
data: {
_task: JOB_TASK.RUN_BATCH,
id: batch.id
},
eta: moment()
});
DefaultLogger.info({ job, eta }, 'The job has been created!');
And here is the code for our Queue class
class HippoQueue {
public queue: Queue;
constructor() {
// Queue connection
const connection: ConnectionOptions = {
host: HIP_BULL_REDIS_HOST,
port: HIP_BULL_REDIS_PORT,
password: HIP_BULL_REDIS_PASS || undefined,
db: HIP_BULL_REDIS_DB
};
this.queue = new Queue(HIP_BULL_QUEUE, {
connection: connection
});
this.queue.on('error', (err: Error) => {
Logger.error(err);
});
}
public async add(task: TaskOptions): Promise<Job> {
// Construct the job options
let delay = 0;
if (task.eta) {
delay = task.eta.valueOf() - Date.now();
}
const opts: JobsOptions = {
delay: delay
};
if (task.attempts) {
opts.attempts = task.attempts;
}
if (task.backoff) {
opts.backoff = task.backoff;
}
if (task.removeOnComplete != null) {
opts.removeOnComplete = task.removeOnComplete;
} else {
opts.removeOnComplete = QUEUE_HISTORY ? false : true;
}
const job = await this.queue.add(task.category, task.data, opts);
// Record the task in the database
try {
const record = new Task({
jobId: job.id,
category: task.category,
data: task.data,
eta: task.eta ? task.eta.toDate() : null,
attempts: opts.attempts
});
await record.save();
} catch (err) {
Logger.error(err);
}
return job;
}
public async get(id: string): Promise<Job | undefined> {
const job = await this.queue.getJob(id);
return job;
}
public async remove(id: string): Promise<Job | undefined> {
const job = await this.queue.getJob(id);
if (job) {
await job.remove();
}
return job;
}
}
const Instance = new HippoQueue();
export default Instance;
Relevant log output
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (11 by maintainers)
Glad that you managed to solve the issue. Btw, try to upgrade your BullMQ version as we have made some important bugfixes since 5.40.0.