sidekiq-unique-jobs: Timed out after 0s while waiting for primed token

Describe the bug When executing some workers we are getting a Timed out after 0s while waiting for primed token warning.

Expected behavior Workers should be executed appropriately.

Current behavior It appears that jobs are started, the warning is thrown, no worker code is executed, and then the job finishes without executing.

Worker class

class PublishWorker
  include Sidekiq::Worker

  sidekiq_options lock: :until_and_while_executing
  # This lock allows one instance of this job with a particuar argument
  # to be in the queue and one to be executing.

  def perform(live_object)
    return if cancelled?

    FeedService::Publish.call(live_object)
  end

  def cancelled?
    REDIS.exists?("sidekiq-cancelled-#{jid}")
  end

  def self.cancel!(jid)
    REDIS.setex("sidekiq-cancelled-#{jid}", 86_400, 1)
  end
end

Additional context This warning and lack of execution is only happen sometimes but not always. Inside of FeedService::Publish.call(live_object) we created nested workers from an iteration. The number of nested workers range from 1~20. They are also until_and_while_executing locks. Usually what happens is the first few nested workers will execute and the rest will not perform. Sometimes even the “top level” worker above however does not perform.

Configuration:

        ::Sidekiq.configure_client do |config|
          config.client_middleware do |chain|
            chain.add SidekiqUniqueJobs::Middleware::Client
            chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
          end
        end

         ::Sidekiq.configure_server do |config|
          config.client_middleware do |chain|
            chain.add SidekiqUniqueJobs::Middleware::Client
            chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
          end

          config.server_middleware do |chain|
            chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
            chain.add SidekiqUniqueJobs::Middleware::Server
            chain.add Sidekiq::Middleware::Server::Statsd
          end

          SidekiqUniqueJobs::Server.configure(config)
        end

version 7.0.9

Any pointers or thoughts would be immensely appreciated! Thank you for your hard work on this spectacular gem!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 41 (18 by maintainers)

Most upvoted comments

Just ran another test where I enqueued 100 jobs that each updated their own database record. In every case where the warning was received, the database record was not updated meaning the job itself didn’t run. This does not seem to be an ignorable warning.

Released as v7.0.12

Sorry I never circled back on this issue. Upgrading to the version v7.0.10 worked for us at least. We still receive the warnings, but our execution occured as expected. Not the most helpful I know for others, but I thought I’d say something for reference.

@monsha @jerryjohnjacob Thanks for the details. I’ll do some experiments this week.

@joshhubers That is super helpful feedback! Thank you

@jerryjohnjacob thanks for the report. I will do some more debugging on the issue given your test example.

It might help in certain scenarios. It depends on how many duplicates are coming in at the same time etc.

In your example above you’d just slow things down a little.

There is conflict resolution you can configure and such things.