sidekiq-unique-jobs: Timed out after 0s while waiting for primed token
Describe the bug
When executing some workers we are getting a Timed out after 0s while waiting for primed token
warning.
Expected behavior Workers should be executed appropriately.
Current behavior It appears that jobs are started, the warning is thrown, no worker code is executed, and then the job finishes without executing.
Worker class
class PublishWorker
include Sidekiq::Worker
sidekiq_options lock: :until_and_while_executing
# This lock allows one instance of this job with a particuar argument
# to be in the queue and one to be executing.
def perform(live_object)
return if cancelled?
FeedService::Publish.call(live_object)
end
def cancelled?
REDIS.exists?("sidekiq-cancelled-#{jid}")
end
def self.cancel!(jid)
REDIS.setex("sidekiq-cancelled-#{jid}", 86_400, 1)
end
end
Additional context
This warning and lack of execution is only happen sometimes but not always. Inside of FeedService::Publish.call(live_object)
we created nested workers from an iteration. The number of nested workers range from 1~20. They are also until_and_while_executing
locks. Usually what happens is the first few nested workers will execute and the rest will not perform. Sometimes even the “top level” worker above however does not perform.
Configuration:
::Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add SidekiqUniqueJobs::Middleware::Client
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
end
end
::Sidekiq.configure_server do |config|
config.client_middleware do |chain|
chain.add SidekiqUniqueJobs::Middleware::Client
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
end
config.server_middleware do |chain|
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
chain.add SidekiqUniqueJobs::Middleware::Server
chain.add Sidekiq::Middleware::Server::Statsd
end
SidekiqUniqueJobs::Server.configure(config)
end
version 7.0.9
Any pointers or thoughts would be immensely appreciated! Thank you for your hard work on this spectacular gem!
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 41 (18 by maintainers)
Just ran another test where I enqueued 100 jobs that each updated their own database record. In every case where the warning was received, the database record was not updated meaning the job itself didn’t run. This does not seem to be an ignorable warning.
Released as v7.0.12
Sorry I never circled back on this issue. Upgrading to the version
v7.0.10
worked for us at least. We still receive the warnings, but our execution occured as expected. Not the most helpful I know for others, but I thought I’d say something for reference.https://github.com/mhenrixon/sidekiq-unique-jobs/issues/571#issuecomment-833699420
@monsha @jerryjohnjacob Thanks for the details. I’ll do some experiments this week.
@joshhubers That is super helpful feedback! Thank you
@jerryjohnjacob thanks for the report. I will do some more debugging on the issue given your test example.
It might help in certain scenarios. It depends on how many duplicates are coming in at the same time etc.
In your example above you’d just slow things down a little.
There is conflict resolution you can configure and such things.