bullmq: [Bug]: Worker stopped processing jobs, and mostly delayed Jobs

Version

v5.4.2

Platform

NodeJS

What happened?

We have a service, where a worker runs, and processes jobs. After the processing is done, it will create another job, which is delayed (around 64 minutes). Today, I noticed that the service and worker stopped processing jobs. There were no error messages in the logs. When I used BullBoard (I use it as a UI to see jobs), I saw the jobs were still in the delayed state, and like 24 hours overdue.

When I restarted the service, and the worker started, it immediately started processing those delayed jobs. This is not the first it happened. Today I though first checked the delayed jobs.

In today’s incident, the service has been running for 4 days.

We run in EKS on AWS (NodeJS service, using Typescript). I use BullMQ Pro. And we are using Groups and each Group has a concurrency set to 1.

How to reproduce.

I don’t have any test code for this

Relevant log output

No Logs or error logs were produced

Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 119 (45 by maintainers)

Commits related to this issue

Most upvoted comments

Since it seems that the original authors cannot reproduce the issue anymore, I will close it now, so that other users do not get lured into this one if they are experiencing a similar issue, but not this exact one, as that will just confuse everybody.

@tavindev you are welcome to open a new issue with the particular details for your use case.

It is not enough adding the line with the disconnect, you must remove the other 2 as well.

@lukas-becker0 seems like I am able to reproduce it following your instructions. I will keep you updated…

Hi, @manast I did some tests with the code you provided.

If I force close the redis connections with e.g. redis-cli CLIENT KILL TYPE normal && redis-cli CLIENT KILL TYPE slave && redis-cli CLIENT KILL TYPE pubsub the worker reconnects and continues to process jobs as expected.

If however I kill/shutdown the entire redis server docker-compose down, then wait a few seconds (~5 seconds) and then start redis again (docker-compose up -d), the worker reconnects but does not continue to process jobs (although isRunning returns true) from the queue. The queue does then no longer see any connected worker.

(Simulating a server restart and/or crash)

Is that the expected behaviour ?

docker-compose.yaml

version: '3.8'

services:
    redis:
        container_name: 'redis_test_server'
        image: redis:6.2.14-alpine
        restart: always
        ports:
            - '6379:6379'
        volumes:
            -   /tmp/redis-test-server:/data

(Setting redis-cli config set maxmemory-policy noeviction after the first start ofcourse)

I modified your code slightly:

const { Queue, Worker } = require('bullmq');

const queueName = 'test';

async function start() {
	const queue = new Queue(queueName, {
		connection: { host: 'localhost', port: 6379 },
		// a warning is thrown on redis startup if these aren't added
		enableReadyCheck: false,
		maxRetriesPerRequest: null,
		enableOfflineQueue: false,
	});

	setInterval(() => {
		queue.getWorkersCount().then((numberOfWorkers) => {
			console.warn(`Number of workers: ${numberOfWorkers}`);
		});

		queue.getJobCounts().then((numberOfJobs) => {
			console.warn(`Number of jobs: ${JSON.stringify(numberOfJobs)}`);
		});
	}, 10_000);

	const job = await queue.add('__default__', null, {
		jobId: queueName + '-cron-worker-job',
		repeat: {
			every: 3000, // every 3 seconds
		},
		data: {
			foo: 'bar',
		},
	});

	const processFn = async (job) => {
		console.log(`Processing job ${job.id} with data ${job.data}`);
		console.log(`-> ${job.id}`);
		await new Promise((res) => setTimeout(res, 1000));
		console.log(`\t<- ${job.id}`);
	};

	const worker = new Worker(queueName, processFn, {
		connection: {
			host: 'localhost',
			port: 6379, // a warning is thrown on redis startup if these aren't added
			enableReadyCheck: false,
			maxRetriesPerRequest: null,
			enableOfflineQueue: true,
		},
	});

	worker.on('error', (err) => {
		console.error(err);
	});

	worker.on('closed', () => {
		console.warn('Worker closed');
	});

	worker.on('ready', () => {
		console.warn('Worker is ready!');
	});

	worker.on('completed', (job) => {
		console.log(`Job ${job.id} completed`);
	});

	worker.on('failed', (job, err) => {
		console.error(`Job ${job.id} failed with ${err.message}`);
	});
}

start();

I am reopening as it seems this is still not working, until we can reproduce it.

We rolled back to 5.4.0 two weeks ago and have not experienced this issue since. Still curious as to the resolution here so we know when it may be safe to attempt to keep BullMQ up-to-date again. Thanks for all your support thus far.

5.7.6 fixes it for us

Unfortunately we are still experiencing the issue with the latest version. Could you please let me know which version introduced this issue so we can downgrade to that specific version.