bullmq: [Bug]: Job goes to delayed status upon creation

Version

v5.40.0

Platform

NodeJS

What happened?

We have been experiencing some latency for our worker. It appears that it needs time to warm up, but it is unclear why this is happening. I thought that it was related to this fix that you guys had made, but the issue persists after upgrading from 5.35.0 to 5.40.0

I am using bullmq v5.40.0

Here is a screenshot of the two jobs being moved to delayed status, even though their delay is set to 0 (sometimes a negative number) Image

I have tried the following:

  • Upgrade to latest version of bullmq
  • Manually promote the job, when I do this the job moves to waiting status for the same amount of time as it would have stayed in the delayed status

According to the docs here, if there is an error and backoff strategy is when adding the job then it goes to delayed status, but how can I visualize that to diagnose what the root cause is?

Image

How to reproduce.

Here is the code for when we instantiate our worker. The worker lives in an Azure App Service

// Start the worker
    this.worker = new Worker(HIP_BULL_QUEUE, this.process, {
      name: 'hippo-worker',
      concurrency: 5,
      connection: {
        host: HIP_BULL_REDIS_HOST,
        port: HIP_BULL_REDIS_PORT,
        password: HIP_BULL_REDIS_PASS,
        db: HIP_BULL_REDIS_DB
      },
      metrics: {
        maxDataPoints: MetricsTime.TWO_WEEKS
      }
    });

Here is the code to add the job

const job = await Queue.add({
    category: JOB_CATEGORY,
    data: {
      _task: JOB_TASK.RUN_BATCH,
      id: batch.id
    },
    eta: moment()
  });
  DefaultLogger.info({ job, eta }, 'The job has been created!');

And here is the code for our Queue class

class HippoQueue {
  public queue: Queue;

  constructor() {
    // Queue connection
    const connection: ConnectionOptions = {
      host: HIP_BULL_REDIS_HOST,
      port: HIP_BULL_REDIS_PORT,
      password: HIP_BULL_REDIS_PASS || undefined,
      db: HIP_BULL_REDIS_DB
    };
    this.queue = new Queue(HIP_BULL_QUEUE, {
      connection: connection
    });
    this.queue.on('error', (err: Error) => {
      Logger.error(err);
    });
  }

  public async add(task: TaskOptions): Promise<Job> {
    // Construct the job options
    let delay = 0;
    if (task.eta) {
      delay = task.eta.valueOf() - Date.now();
    }
    const opts: JobsOptions = {
      delay: delay
    };
    if (task.attempts) {
      opts.attempts = task.attempts;
    }
    if (task.backoff) {
      opts.backoff = task.backoff;
    }
    if (task.removeOnComplete != null) {
      opts.removeOnComplete = task.removeOnComplete;
    } else {
      opts.removeOnComplete = QUEUE_HISTORY ? false : true;
    }
    const job = await this.queue.add(task.category, task.data, opts);
    // Record the task in the database
    try {
      const record = new Task({
        jobId: job.id,
        category: task.category,
        data: task.data,
        eta: task.eta ? task.eta.toDate() : null,
        attempts: opts.attempts
      });
      await record.save();
    } catch (err) {
      Logger.error(err);
    }
    return job;
  }

  public async get(id: string): Promise<Job | undefined> {
    const job = await this.queue.getJob(id);
    return job;
  }

  public async remove(id: string): Promise<Job | undefined> {
    const job = await this.queue.getJob(id);
    if (job) {
      await job.remove();
    }
    return job;
  }
}

const Instance = new HippoQueue();
export default Instance;

Relevant log output


Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 23 (11 by maintainers)

Most upvoted comments

Glad that you managed to solve the issue. Btw, try to upgrade your BullMQ version as we have made some important bugfixes since 5.40.0.