sidekiq-unique-jobs: Are we meant to manually expire the unique jobs hash?
Hi, thanks for this gem. We use it extensively in some of ours app. Recently we had a bunch of OOM warnings because it turns out the uniquejobs collection grew out of hand and there is no TTL on this key.
I realise that you provide an SidekiqUniqueJobs::Util.expire is this meant to be run periodically? Or what is the best way to ensure the unique jobs hash never grows again like this?
I noticed that for one of our smaller apps this hash gets correctly managed. Is there some race condition? Am i missing some configuration? Is any further info I should debug?
> hlen uniquejobs
(integer) 835783
Thanks
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 2
- Comments: 22 (8 by maintainers)
Commits related to this issue
- update siekiq-unique-jobs again seems there is a fix to remove all orphaned keys in this 6.0.5 release. https://github.com/mhenrixon/sidekiq-unique-jobs/issues/234#issuecomment-411070779 — committed to camallen/Panoptes by camallen 6 years ago
- update siekiq-unique-jobs again (#2879) seems there is a fix to remove all orphaned keys in this 6.0.5 release. https://github.com/mhenrixon/sidekiq-unique-jobs/issues/234#issuecomment-411070779 — committed to zooniverse/panoptes by camallen 6 years ago
@mhenrixon The issue that @thomey described before reoccured. Now all the jods are EXISTS and GRABBED. Any idea why?
Ah, I see what you mean @thomey. Right now the gem is signaling to release the locks but they are kept around unless expiration is set. I don’t recommend you use
lock_expiration. I’ll see what is involved in hard deleting the locks instead. I guess the lua script should delete the keys if no expiration is set and set expiration on the keys if expiration is provided.While I’m at it I will create a script to find and delete orphaned keys also so that you don’t have to manually do that.
It appears the
Util.expirewas removed–is there a better solution to clearing the keys?I ran into this same issue except we have 23M expired keys and growing. I created a Sidekiq job to delete the expired keys 2k at a time: https://gist.github.com/dennisfaust/c94f0e1aec54e37c52e431e9542ff042 There are a couple of things like the particular Redis DB that are unique to our situation but I’m sure you can modify it. I’m also upgrading to the latest gem in hopes the hash won’t keep growing.