sidekiq-unique-jobs: Failed jobs waiting to be retried are not considered when fetching uniqueness

Describe the bug If a job fails, and is waiting to be retried, adding a new job with similar arguments should trigger the conflict resolution mechanism.

I have an job which - based on its arguments - calculates and saves an expensive operation to the database, that I would not like to enqueue multiple times (since it’s output is only based on the input, and I want to make the calculation as few times as possible…), but if it is already being executed, allow it t be enqueued. Curently I am using this config:

    sidekiq_options lock: :until_executing, on_conflict: :replace

   # only use the id for uniqueness checking, and ignore the actual data to be aggregated
    def self.unique_args(args)
      [ args[0] ]
    end

until_executed is not okay due to the requirement for it to be reexecuted if it is enqueued while it is already running.

Now the issue here is that the lock is removed when the job is scheduled to be run, but it is not readded if it fails.

Expected behavior The system should re-add the lock if the job fails, and trigger the deduplication mechanism if a similar job has been enqueued in the meantime. If you think about it this is not very different than adding a job to the end of the enqueued jobs list, other than it contains some meta-information about the failure, and how many times it was tried. Not only does it get duplicated, but the execution order is undeterministic, sometimes the job that is scheduled later will be executed first, and when the job from the retry queue is rescheduled, it is executed as well, overwriting the fresh data with old one.

Current behavior Jobs get duplicated