sidekiq-unique-jobs: Jobs executing and immediately returning

Describe the bug We’ve had two instances in which we were using sidekiq_options unique: :until_and_while_executing on a worker and after some time, calling .perform_async on the worker would result in the job getting enqueued, a job ID assigned, picked up by a worker, and then exit immediately.

For example, we would see in the sidekiq.log

2019-09-09T14:52:16.545Z 14236 TID-ovo800gss MyWorker JID-3287686b0cad75fee25a57c8 INFO: start
2019-09-09T14:52:16.548Z 14236 TID-ovo800gss MyWorker JID-3287686b0cad75fee25a57c8 INFO: done: 0.003 sec

In all cases, there were no other instances of MyWorker enqueued or running.

Upon changing to sidekiq_options unique: :until_executing the problem went away in both cases.

Expected behavior The worker to execute as normal.

Current behavior See above. Note in the class below we are logging start, and success or failure - in these cases we see nothing logged. These are also typically workers that run for several minutes.

Worker class

class MyWorker
  include Sidekiq::Worker

  sidekiq_options queue: :data_sync, retry: true, backtrace: true
  sidekiq_options unique: :until_and_while_executing

  def perform(some_id)
    some_model = SomeModel.find(some_id)

    log('Started', some_model)
    result = some_model.sync.call

    if result.success?
      log('Finished', some_model)
    else
      log("Failed: #{result.errors.join(', ')}", some_model)
    end
  end

  private

  def log(message, some_model = nil)
    EventLogger.log(
      :sync,
      note: message,
      loggable: some_model
    )
  end
end

Additional context Anything we can look for here? Ideally we’d want to use :until_and_while_executing if possible

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 23 (10 by maintainers)

Most upvoted comments

I am running on 7.0.0beta22 and got this error:

until_and_while_executing=uniquejobs:6579078eee7b12eb46e1a08399196e83 WARN: Timed out after 0s while waiting for primed token (digest: uniquejobs:6579078eee7b12eb46e1a08399196e83, job_id: 20b5aa8952067bb50e007265)

then nothing happened, no job entered the queue

UPDATE: I could reproduce it locally, the deadlocks are left on server restarts and are never released. Adding the lock_ttl in the config ensures that the dead locks will expire after a period of time. Also in the readme the section https://github.com/mhenrixon/sidekiq-unique-jobs#cleanup-dead-locks doesn’t exactly work: delete_by_digest is an instance method so it needs to be: SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest and I could never get digest to be present. I don’t know exactly why.

@mhenrixon I was finally able to upgrade this and simultaneously replace ActiveJob with Sidekiq::Worker (in most places) about a month ago. So far it’s been smooth sailing. Thanks for your hard work on this!

Good to know. Also maybe some consideration into shutdown procedure. I scaled down to a single process, and I thought I was ok, but as I re-deployed, there’s technically 2 running at the same time for a small amount of time. It would be nice if this also didn’t cause the kaboom.