Hangfire.Mongo: Jobs Stuck in Enqueued State

First posted on GitHub/HangfireIO (#2355)

I can provide any additional information that you require to help troubleshoot this with me.

Issue

Jobs are staying/stuck in the Enqueued state. The system thinks those jobs are still running and won’t enqueue them again. So in the instance from the screen shot, we have 63 unique recurring jobs that never get enqueued again. The only way I can find to get them running again is to stop the app pool, drop all hangfire.* collections from mongo, and then start the app pool again. (we add the recurring jobs on startup)

image

Solution State

ASP.NET Core .NET 8 Hangfire.AspNetCore Version=“1.8.9” Hangfire.Console Version=“1.4.2” Hangfire.Core Version=“1.8.9” Hangfire.Dashboard.BasicAuthorization Version=“1.0.2” Hangfire.Mongo Version=“1.9.16”

The most recent stdump file: stdump_hangfire2.txt

Classes have this attribute applied: SkipWhenPreviousJobIsRunningAttribute.txt

Execute Methods have [DisableConcurrentExecution("{0}", 3)] applied

image

image

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 51 (25 by maintainers)

Commits related to this issue

Most upvoted comments

@mwasson74,

Lets close it for now. We will reopen if something else comes up. Thank you for the generous contribution!

Great news!

The root bug was that I did not properly set job states when job instance was requeued while processing. It was triggered by someone on your end requeuing it from the dashboard even though the job was still running. This enqueued the same job again, but as I had not set the required fields, the job was stuck in “Enqueued” state which was also reflected in the dashboard. This in combination with the DisableConcurrentExection filter which caused a deadlock. (only when requeuing an already running job)

Triggering an already running job is allowed by the hangfire dashboard. the user will, however loose job history for the first running instance of the job as the second’s state history will be written instead as they both have the same JobId. Triggering from the “Recurring jobs” page will create a new instance (with new JobId). This also happens in Hangfire.SqlStorage.

When a running job is requeued, the job history will look like:

  1. “Enqueued”
  2. “Processing”
  3. “Enqueued” (by dashboard user)
  4. “Processing”
  5. “Succeeded”

The first two state belongs the first job which was started by hangfire by its CRON the last three belongs the second job which was started by the user. Whatever happens to the first job (succeeds, fails) is lost.

SlidingInvisibilityTimeout would with default values mark a job stale after 5 minutes without heartbeat (Hangfire.Mongo will send heartbeats x5 by given timeout), allowing hangfire to start the job in a different worker. This obsoletes the InvisibilityTimeout which was a fixed value ALL jobs was expected to run to completion within. With SlidingInvisibilityTimeout the job can take as long as needed and will not be marked stale as long as it has a heartbeat.

its a little complex to explain, but I hope it makes sense.

Thanks!

I dit think so. Hangfire.Mongo now behaves exactly like Hangfire.SqlStorage. One thing, which applies to both libraries is that if the job is in fact processing when requeued and not stale, the state of the first job will be lost as the second enqueued job will override the states.

Im not sure if it makes sense to handle this case though as it will only happen if a user requeued it from dashboard even though it was processing.

Damnit! I thought we had it.

Off course I’ll take a look! We need to figure this out! 🙂

Nothing in the log?

Thanks

its released, it should be indexed in nuget.org shortly.

Man, thank you so much!! 🙏

@mwasson74,

$$NOW is an aggregation pipeline variable, with double $. There might something wrong with my tests, as I did test this against v4.0.13. 😕

Anyway, I will revert this part as it is a little bit of a hack query-wise. At a later stage I will create a new release with focus on using mongodb server time instead of Datetime.UtcNow

Give me 20m. then I have a v1.10.2 with this part reverted

Got it. thanks. I will spin up a docker and just sanity check it.

@gottscj

That was fast, thank you!! I can tell you as long as you don’t make fun of me/us 😬 It’s 4.0.13

@mwasson74,

Yes definitely. I will patch it tonight.

Thanks for the feedback!

@gottscj

I just deployed our solution with the new update you pushed and so far things are ok but we are getting the below exception many times through the job running. A quick Google shows me this SO answer and it is true, we are using an older version of Mongo. Is there any way that you could use $addFields instead of $set in the code here (I assume) since it’s supposed to work with older versions? MongoDistributedLock.cs#L295

Hangfire:ftusbridge_mugshotftps_euus72_mugshot - Unable to update heartbeat on the resource. Details:
MongoDB.Driver.MongoCommandException: Command aggregate failed: Unrecognized pipeline stage name: '$set'.
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, ICoreSession session, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadOperationExecutor.Execute[TResult](IRetryableReadOperation`1 operation, RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.Execute(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.Execute(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.AggregateOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.Aggregate[TResult](IClientSessionHandle session, PipelineDefinition`2 pipeline, AggregateOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass22_0`1.<Aggregate>b__0(IClientSessionHandle session)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at Hangfire.Mongo.DistributedLock.MongoDistributedLock.<StartHeartBeat>b__16_0(Object _)

@mwasson74,

The InvisibilityTimeout, has been replaced with SlidingInvisibilityTimeout, which is set to a default value of 5m. So you can just remove the line

InvisibilityTimeout = TimeSpan.FromMinutes(30)

@mwasson74,

Looking at the jobs in the DB, it seems they were all “Processing” when you requeued them from the dashboard.

image

I have found a bug in Hangfire.Mongo where, changing the state (from processing to Enqueued) in UI might will cause the job not to be requeued again, as it has lost its queue.

I’m currently working on a fix.

In your case, its probably more a visual thing as enqueueing a “Processing” job would ultimately cause it to be cancelled anyway due to your filter.

Requeuing a “Processing” job will not cause it restart, it will just spawn a new job which ultimately would run concurrently with the already running job. There is currently no way of stopping or deleting an already running job. Deleting it, will only set the job state. the actual worker will still run to completion. It does look like the SqlStorage provided with the Hangfire would suffer the same behavior, I will try to reproduce using the SqlStorage.

For now, I recommend not re-queueing “Processing” jobs using the dashboard. You could delete it and then trigger it again from the recurring jobs page. This would most likely also be what you wanted in the first place? This would however override your SkipWhenPreviousJobIsRunningAttribute attribute

Thanks Jonas

@mwasson74,

thank you for the file. I’m struggling to consume it. Could you guide me how to restore the given file?

Thanks!

When this happened again the other day, I had exactly what my initial screen shot shows: Enqueued 63/0 with nothing processing. I then exported all hangfire.* collections before getting production up and running again by dropping the collections and restarting the app.

I restored hangfire.jobGraph to my local instance of mongo, sanitized what I could find 🤞 and started writing queries.

When I try running a version of your suggested query, only the first part of the or-statement comes back (76 documents) because there aren’t any where StateName is set to “Processing”: db.getCollection("hangfire.jobGraph").find({ $or: [{ Key: { $regex: /^recurring-job:/i } }, { _t: "JobDto", StateName: "Processing" }] })

When I remove StateName from the query, I get 19,024 documents: db.getCollection("hangfire.jobGraph").find({ $or: [{ Key: { $regex: /^recurring-job:/i } }, { _t: "JobDto" }] })

I’ve exported the whole sanitized collection (not just exporting query results) and attached it here. I exported it as a .agz and then changed the extension to .zip so I could upload it.

hangfire.jobGraph.sanitized.zip

I hope you can find something with all of this data!! 🤞

Thanks again!!