sidekiq-unique-jobs: :until_executed jobs get stuck every now and then

Describe the bug New unique jobs can’t be added to the queue despite the last one being complete and no other similar job being in the queue. This seems to be related to the keys not being properly cleared.

The jobs in question are from the CRON and ran every minute. The issue happens at random; sometimes spaced by a week, sometimes multiple times in a day.

Our way to go around this is to wipe out all unique keys when this happens.

Worker class

  sidekiq_options queue: :critical, retry: 0, unique: :until_executed

Sidekiq conf

Sidekiq.configure_server do |config|
  config.failures_default_mode = :exhausted
  config.redis = REDIS_MAIN_CONFIG.merge({ size: sidekiq_redis_size, namespace: "sidekiq:hacksterio", network_timeout: 8 })

  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 1.minute
  end
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end

  config.death_handlers << ->(job, _ex) do
    SidekiqUniqueJobs::Digests.del(digest: job['unique_digest']) if job['unique_digest']
  end

  if defined?(ActiveRecord::Base)
    Rails.logger.debug("Setting custom connection pool size of #{sidekiq_pool} for Sidekiq Server")
    config = ActiveRecord::Base.configurations[Rails.env] ||
                Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = (ENV['DB_REAP_FREQ'] || 10).to_i  # seconds
    config['pool']              = sidekiq_pool + 2  # buffer
    ActiveRecord::Base.establish_connection(config)

    Rails.logger.info("Connection pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('@size')}")
  end
end

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 5
Comments: 25 (14 by maintainers)

Most upvoted comments

@soma @simonoff @ccleung @blarralde I am working on fixing it once and for all. It is turning out to be quite an endeavor but i am getting there ever so slowly.

mhenrixon on May 3, 2019

I am going to close this issue. It has been fixed on master (version 7) that no locks will keep being locked forever.

@mhenrixon Could you post a reference to the commit / PR that fixed this issue?

dbackeus on Jan 16, 2020

@mhenrixon - just to double check - would implementing something like this help with the issue we have here? https://github.com/mhenrixon/sidekiq-unique-jobs#cleanup-dead-locks

On another note - apparently one of my colleagues downgraded to 6.0.8 and the problem hasn’t manifested for their project anymore - could this have been an issue introduced between that version and the latest?

ccleung on Mar 15, 2019

Here’s a quick and dirty garbage collector, let me know what you think:

module SidekiqUniqueGarbageCollector
  extend self

  def clean_unused
    unused_digests.each do |digest|
      SidekiqUniqueJobs::Digests.del digest: digest
    end
  end

  private
    def all_active_digests
      busy_digests + queued_digests
    end

    def all_digests
      SidekiqUniqueJobs::Digests.all.map{|d| d.gsub(SIDEKIQ_NAMESPACE, '')}
    end

    def all_queues
      [
        Sidekiq::RetrySet.new,
        Sidekiq::ScheduledSet.new,
      ] + named_queues
    end

    def busy_digests
      Sidekiq::Workers.new.map{|process, thread, msg| Sidekiq::Job.new(msg['payload'])['unique_digest'] }
    end

    def named_queues
      queue_names.map{|name| Sidekiq::Queue.new(name) }
    end

    def queued_digests
      all_queues.map{|queue| queue.map{|job| job['unique_digest'] } }.flatten.compact
    end

    def queue_names
      Sidekiq::Queue.all.map(&:name)
    end

    def unused_digests
      all_digests - all_active_digests
    end
end

blarralde on Mar 8, 2019