Hangfire.Mongo: Jobs Stuck in Enqueued State
First posted on GitHub/HangfireIO (#2355)
I can provide any additional information that you require to help troubleshoot this with me.
Issue
Jobs are staying/stuck in the Enqueued state. The system thinks those jobs are still running and won’t enqueue them again. So in the instance from the screen shot, we have 63 unique recurring jobs that never get enqueued again. The only way I can find to get them running again is to stop the app pool, drop all hangfire.* collections from mongo, and then start the app pool again. (we add the recurring jobs on startup)
Solution State
ASP.NET Core .NET 8 Hangfire.AspNetCore Version=“1.8.9” Hangfire.Console Version=“1.4.2” Hangfire.Core Version=“1.8.9” Hangfire.Dashboard.BasicAuthorization Version=“1.0.2” Hangfire.Mongo Version=“1.9.16”
The most recent stdump file: stdump_hangfire2.txt
Classes have this attribute applied: SkipWhenPreviousJobIsRunningAttribute.txt
Execute Methods have [DisableConcurrentExecution("{0}", 3)] applied
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 51 (25 by maintainers)
Commits related to this issue
- Fix job not enqueued when requeued while processing #380 - Change strategy of how to determine whether a background job that a worker dequeued is still alive and being processed - Using same strategy... — committed to gottscj/Hangfire.Mongo by gottscj 3 years ago
- Sliding timeout and fix jobs not enqueued when requeued while processing (#380) (#381) * Fix job not enqueued when requeued while processing #380 - Change strategy of how to determine whether a ba... — committed to gottscj/Hangfire.Mongo by gottscj 3 years ago
@mwasson74,
Lets close it for now. We will reopen if something else comes up. Thank you for the generous contribution!
Great news!
The root bug was that I did not properly set job states when job instance was requeued while processing. It was triggered by someone on your end requeuing it from the dashboard even though the job was still running. This enqueued the same job again, but as I had not set the required fields, the job was stuck in “Enqueued” state which was also reflected in the dashboard. This in combination with the DisableConcurrentExection filter which caused a deadlock. (only when requeuing an already running job)
Triggering an already running job is allowed by the hangfire dashboard. the user will, however loose job history for the first running instance of the job as the second’s state history will be written instead as they both have the same JobId. Triggering from the “Recurring jobs” page will create a new instance (with new JobId). This also happens in Hangfire.SqlStorage.
When a running job is requeued, the job history will look like:
The first two state belongs the first job which was started by hangfire by its CRON the last three belongs the second job which was started by the user. Whatever happens to the first job (succeeds, fails) is lost.
SlidingInvisibilityTimeout would with default values mark a job stale after 5 minutes without heartbeat (Hangfire.Mongo will send heartbeats x5 by given timeout), allowing hangfire to start the job in a different worker. This obsoletes the InvisibilityTimeout which was a fixed value ALL jobs was expected to run to completion within. With SlidingInvisibilityTimeout the job can take as long as needed and will not be marked stale as long as it has a heartbeat.
its a little complex to explain, but I hope it makes sense.
Thanks!
I dit think so. Hangfire.Mongo now behaves exactly like Hangfire.SqlStorage. One thing, which applies to both libraries is that if the job is in fact processing when requeued and not stale, the state of the first job will be lost as the second enqueued job will override the states.
Im not sure if it makes sense to handle this case though as it will only happen if a user requeued it from dashboard even though it was processing.
Damnit! I thought we had it.
Off course I’ll take a look! We need to figure this out! 🙂
Nothing in the log?
Thanks
its released, it should be indexed in nuget.org shortly.
Man, thank you so much!! 🙏
@mwasson74,
$$NOW is an aggregation pipeline variable, with double $. There might something wrong with my tests, as I did test this against v4.0.13. 😕
Anyway, I will revert this part as it is a little bit of a hack query-wise. At a later stage I will create a new release with focus on using mongodb server time instead of Datetime.UtcNow
Give me 20m. then I have a v1.10.2 with this part reverted
Got it. thanks. I will spin up a docker and just sanity check it.
@gottscj
That was fast, thank you!! I can tell you as long as you don’t make fun of me/us 😬 It’s 4.0.13
@mwasson74,
Yes definitely. I will patch it tonight.
Thanks for the feedback!
@gottscj
I just deployed our solution with the new update you pushed and so far things are ok but we are getting the below exception many times through the job running. A quick Google shows me this SO answer and it is true, we are using an older version of Mongo. Is there any way that you could use
$addFieldsinstead of$setin the code here (I assume) since it’s supposed to work with older versions? MongoDistributedLock.cs#L295@mwasson74,
The InvisibilityTimeout, has been replaced with SlidingInvisibilityTimeout, which is set to a default value of 5m. So you can just remove the line
InvisibilityTimeout = TimeSpan.FromMinutes(30)@mwasson74,
Looking at the jobs in the DB, it seems they were all “Processing” when you requeued them from the dashboard.
I have found a bug in Hangfire.Mongo where, changing the state (from processing to Enqueued) in UI might will cause the job not to be requeued again, as it has lost its queue.
I’m currently working on a fix.
In your case, its probably more a visual thing as enqueueing a “Processing” job would ultimately cause it to be cancelled anyway due to your filter.
Requeuing a “Processing” job will not cause it restart, it will just spawn a new job which ultimately would run concurrently with the already running job. There is currently no way of stopping or deleting an already running job. Deleting it, will only set the job state. the actual worker will still run to completion. It does look like the SqlStorage provided with the Hangfire would suffer the same behavior, I will try to reproduce using the SqlStorage.
For now, I recommend not re-queueing “Processing” jobs using the dashboard. You could delete it and then trigger it again from the recurring jobs page. This would most likely also be what you wanted in the first place? This would however override your SkipWhenPreviousJobIsRunningAttribute attribute
Thanks Jonas
@mwasson74,
thank you for the file. I’m struggling to consume it. Could you guide me how to restore the given file?
Thanks!
When this happened again the other day, I had exactly what my initial screen shot shows: Enqueued 63/0 with nothing processing. I then exported all hangfire.* collections before getting production up and running again by dropping the collections and restarting the app.
I restored hangfire.jobGraph to my local instance of mongo, sanitized what I could find 🤞 and started writing queries.
When I try running a version of your suggested query, only the first part of the or-statement comes back (76 documents) because there aren’t any where StateName is set to “Processing”:
db.getCollection("hangfire.jobGraph").find({ $or: [{ Key: { $regex: /^recurring-job:/i } }, { _t: "JobDto", StateName: "Processing" }] })When I remove StateName from the query, I get 19,024 documents:
db.getCollection("hangfire.jobGraph").find({ $or: [{ Key: { $regex: /^recurring-job:/i } }, { _t: "JobDto" }] })I’ve exported the whole sanitized collection (not just exporting query results) and attached it here. I exported it as a .agz and then changed the extension to .zip so I could upload it.
hangfire.jobGraph.sanitized.zip
I hope you can find something with all of this data!! 🤞
Thanks again!!