sidekiq-unique-jobs: Jobs executing and immediately returning
Describe the bug
We’ve had two instances in which we were using sidekiq_options unique: :until_and_while_executing
on a worker and after some time, calling .perform_async
on the worker would result in the job getting enqueued, a job ID assigned, picked up by a worker, and then exit immediately.
For example, we would see in the sidekiq.log
2019-09-09T14:52:16.545Z 14236 TID-ovo800gss MyWorker JID-3287686b0cad75fee25a57c8 INFO: start
2019-09-09T14:52:16.548Z 14236 TID-ovo800gss MyWorker JID-3287686b0cad75fee25a57c8 INFO: done: 0.003 sec
In all cases, there were no other instances of MyWorker
enqueued or running.
Upon changing to sidekiq_options unique: :until_executing
the problem went away in both cases.
Expected behavior The worker to execute as normal.
Current behavior See above. Note in the class below we are logging start, and success or failure - in these cases we see nothing logged. These are also typically workers that run for several minutes.
Worker class
class MyWorker
include Sidekiq::Worker
sidekiq_options queue: :data_sync, retry: true, backtrace: true
sidekiq_options unique: :until_and_while_executing
def perform(some_id)
some_model = SomeModel.find(some_id)
log('Started', some_model)
result = some_model.sync.call
if result.success?
log('Finished', some_model)
else
log("Failed: #{result.errors.join(', ')}", some_model)
end
end
private
def log(message, some_model = nil)
EventLogger.log(
:sync,
note: message,
loggable: some_model
)
end
end
Additional context
Anything we can look for here? Ideally we’d want to use :until_and_while_executing
if possible
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 23 (10 by maintainers)
I am running on 7.0.0beta22 and got this error:
then nothing happened, no job entered the queue
UPDATE: I could reproduce it locally, the deadlocks are left on server restarts and are never released. Adding the
lock_ttl
in the config ensures that the dead locks will expire after a period of time. Also in the readme the section https://github.com/mhenrixon/sidekiq-unique-jobs#cleanup-dead-locks doesn’t exactly work:delete_by_digest
is an instance method so it needs to be:SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
and I could never getdigest
to be present. I don’t know exactly why.@mhenrixon I was finally able to upgrade this and simultaneously replace ActiveJob with Sidekiq::Worker (in most places) about a month ago. So far it’s been smooth sailing. Thanks for your hard work on this!
Good to know. Also maybe some consideration into shutdown procedure. I scaled down to a single process, and I thought I was ok, but as I re-deployed, there’s technically 2 running at the same time for a small amount of time. It would be nice if this also didn’t cause the kaboom.