sidekiq-cron: Unexpected blocking `KEYS` command at 2.0.0
We upgraded sidekiq-cron to version 2.0.0 from 1.12.0, and noticed that at Sidekiq’s startup, it performs a blocking KEYS cron_jobs:* command. This command takes over 5 seconds on large Redis databases, which leads to client timeouts.
Before:
$ redis-cli monitor | grep 'cron_job'
1731416178.107613 [1 [::1]:41712] "smembers" "cron_jobs"
After:
$ redis-cli monitor | grep 'cron_job'
1731416133.431067 [1 [::1]:50116] "keys" "cron_jobs:*"
1731416133.431376 [1 [::1]:50116] "smembers" "cron_jobs:default"
We have two questions regarding this behavior:
- We reviewed the changelog but didn’t see any mention of this change. Local and production-like testing didn’t reveal the problem, as it only occurs with larger Redis installations (over 10 million keys). Should this be highlighted somewhere, especially considering there’s a dedicated warning regarding using this command in production?.
- Is the use of the KEYS wildcard truly necessary?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 36 (29 by maintainers)
Commits related to this issue
- Rework namespaces feature The default for `available_namespaces` is now `[config.default_namespaces]`. See https://github.com/sidekiq-cron/sidekiq-cron/issues/516 — committed to viralpraxis/sidekiq-cron by viralpraxis 2 years ago
- Rework namespaces feature The default for `available_namespaces` is now `[config.default_namespaces]`. See https://github.com/sidekiq-cron/sidekiq-cron/issues/516 — committed to viralpraxis/sidekiq-cron by viralpraxis 2 years ago
I believe the
KEYScommand should be avoided entirely. It’s comparable to Rails performing a full sequential scan (without an index) across an entire relational database. While it might work fine in a local environment or small production setups, it would be a disaster in real-world systems. With Redis, the situation is even worse because the command blocks the entire server.Scanning Redis once (e.g., during startup) is also problematic, as it still blocks the server for seconds and could prevent other clients or gems from initializing. In the best-case scenario, replacing
KEYSwith batchedSCANcould result in Sidekiq being delayed by ~10–inf seconds, depending on the size of the Redis database.Option 2 aligns with how Sidekiq’s queues work, and everything I’ve mentioned explains why it’s implemented this way.
I’ve looked into it, and it seems that using
SCANinstead of KEYS offers limited benefit. WhileSCANprevents Sidekiq from blocking the entire Redis server for [many] seconds, each call toSidekiq::Cron::Namespace.{all,all_with_count}would still take considerable time (basically time forKEYSplus extra latency overhead), making it practically unusable.Note that Sidekiq itself explicitly avoids using this type of query (
O(n), wherenis total keys count) by working with queues specified at configuration time. Iterating over jobs within a specific queue only requires a single lookup in Redis’s queue structure.ATM we will probably just monkeypatch this method like this
but it would be cool to make the entire “namespacing” optional.
There are two steps we can take to mitigate the issue. First, while I completely understand that the
namespaces = conn.keys('cron_jobs:*').collect do |key|part was introduced in version 2.0.0 as an implementation detail, it has a notable side effect of blocking the entire Redis server for a duration proportional to the number of keys. I believe we are not the only Sidekiq-Cron users with a database of this size (which I wouldn’t consider “extremely huge”, just relatively big), so I think it should be mentioned that the gem now performs a full database scan at startup (maybe in release description?). The issue is hard (if not impossible) to catch in staging environments for obvious reasons.Secondly, it might be possible to replace the
KEYScommand withSCAN, which would alleviate some of the blocking issues. I’m not familiar with the codebase, so I’m not yet sure if this approach is applicable here, but I’ll take a look.@taylorrf We were running version 2.0.1. The
KEYScommand was blocking Redis instance and some of requests began timing out. With config update and 2.2.0 you should be fine.@viralpraxis I have not dug into the details of the namespaces feature, but my understanding is that if it is not explicitly configured it will fallback to the default namespace. The same default namespace is configured with available_namespaces and nothing will break. Or will it?
Either way, this is still just a workaround and the main fix should be not to use the keys command at all. From my perspective, namespaces can’t be used now in high-load environments without changin that.
Even official Redis documentation says that keys should be never used in production.
Hi, I’ve just had an issue in production triggered by this keys call.
I’d like to suggest a potential improvement:
KEYScommand usage for most users who don’t explicitly need namespacingSCANinstead ofKEYS: This is a cursor-based approach that avoids blocking the Redis serverThis change would significantly improve the performance and scalability of Sidekiq Cron, particularly for users with extensive Redis deployments.
Hello. Just wanted to +1 and mention that my company is also seeing this issue. We have a dozens of ruby services which use
sidekiq-crongem andkeyscommand is also forbidden by our database ops team for a good reason - as was already mentioned above. So we can’t upgrade to the versions 2 and above.Thank you so much for working on the fix!
If we take this path (moving it to configuration option), I think we should keep anyway the namespacing feature always on, by using the “default” as the default available namespace.
It also remains me how the I18n module works and its
available_localessetting.Maybe it’s the only way to avoid the whole scan? This approach (
available_localeswith fallback to full scan) seems probably fine https://github.com/sidekiq-cron/sidekiq-cron/issues/516#issuecomment-2482731532.I don’t really see how we can fix the things and keep total backwards compatibility. Given that we all agree that performing
KEYSby default is unacceptable here’s the best way to manage this issue I can think of:available_namespacesfromnilto%w[default]available_namespaces = :autowhich will infer available namespaces usingKEYSIt will be backward-incompatible only for people using namespacing and not specifying them explicitly – they’ll have to either specify
:auto(to keep current behaviour) or list all the used namespaces by hand; new users and users not using namespacing will be fine.Honestly, I agree with you that namespacing should be optional and disabled by default. We can achieve this simply by setting
available_namespacesto%w[default]instead ofnilon the gem’s side. The main issue right now is that it’s not entirely clear to users that usingsidekiq-cronwithout explicitly configured namespaces can break apps with large Redis deployments—and that’s very wrong, in my opinion.Hey @eileen-jiang, that’s correct.
That sounds good to me.
So to recap, this gem need a
available_namespacesconfiguration option that:KEYSmethod)The third option is to retain the current behavior if
available_namespacesis not specified (or e.g. set to special value:automatic). This approach would satisfy both users who want it to work automatically and those of us managing large Redis deployments.Yes, I think we have 2 options:
available_namespaces.Not sure which one is better 😅
I can confirm that
KEYS cron_jobs:*is performed each time/sidekiq/cronis accessed.