bullmq: After update to bullmq: Error: could not renew lock for job

Version

v5.34.2

Platform

NodeJS

What happened?

We used the predecessor bull successfully and very heavily over many months at our company. Now, we updated to bullmq and, to be honest, we have quite some issues.

Our queues get stuck quite frequently (we never had this issue!), and we sometimes run into this error:

Error: could not renew lock for job xyz

This just continues like that, it never resolves, until we do a restart. This is quite bad for us.

I also could not really find something here in any other issues? What can we do here?

How to reproduce.

No response

Relevant log output


Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 9
  • Comments: 27 (11 by maintainers)

Most upvoted comments

I am having the ssame issue. It happens randomly and I cannot even destroy the queue. I have to restart each time.

It is highly unlikely that you are having “the same issue”, specially when we do not even know yet what the issue it. So please if you have an issue, post a reproducible case in a new issue and we will look into it.

I am having the ssame issue. It happens randomly and I cannot even destroy the queue. I have to restart each time.

@kosiakMD how do you know it is the same issue? you are getting similar errors, but it is not possible to know if what caused the poster issue is what caused your issue. The most useful thing to do if you want a quick and effective resolution is to create a new issue with all the information you can provide for your particular use case and especially useful a code snippet that reproduces the issue you have.

Quick update from my end:

It looks like that, indeed, we identified some nested for loops and such, that block the event loop. It just took us a very long time to find them. 😬

Will report back when I know (even) more!

@melihplt

  • we use self hosted docker containers on AWS EC2
  • yes, we do add jobs inside the workers
  • actually yes we do. We have Meteor’s DDP connection in place to connect to our main server.

Hello everyone,

No, we do not use typescript, just vanilla JS. The thing is, that we continue to run into this issue. We now even implemented these 2 options for our workers:

{
  maxStalledCount: 0, // do NOT allow to "retry" a stalled job. This CAN lead to a situation, where MULTIPLE workers work on the same job!
  stalledInterval: 1 * 60 * 1000 // 1 minute
}

… and we continue to have this issue. We need to restart the whole node instance (docker container) to make it startup again.

We run into the failed event with the error: Error: could not renew lock for job xyz.

We are already trying to not overload the CPU as best as we can. Of course we “fluctuate around 100%”, but this, to me, does not mean that we really leave ZERO headroom for the CPU to even renew the lock. 😕

Migration:

We are using a different prefix (bullmq). So we kinda discontinued the old bull queue and deployed all our instances in the right order to make the transition as smooth as possible.