sidekiq-cron: Unexpected blocking `KEYS` command at 2.0.0

We upgraded sidekiq-cron to version 2.0.0 from 1.12.0, and noticed that at Sidekiq’s startup, it performs a blocking KEYS cron_jobs:* command. This command takes over 5 seconds on large Redis databases, which leads to client timeouts.

Before:

$ redis-cli monitor | grep 'cron_job'
1731416178.107613 [1 [::1]:41712] "smembers" "cron_jobs"

After:

$ redis-cli monitor | grep 'cron_job'
1731416133.431067 [1 [::1]:50116] "keys" "cron_jobs:*"
1731416133.431376 [1 [::1]:50116] "smembers" "cron_jobs:default"

We have two questions regarding this behavior:

  1. We reviewed the changelog but didn’t see any mention of this change. Local and production-like testing didn’t reveal the problem, as it only occurs with larger Redis installations (over 10 million keys). Should this be highlighted somewhere, especially considering there’s a dedicated warning regarding using this command in production?.
  2. Is the use of the KEYS wildcard truly necessary?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 36 (29 by maintainers)

Commits related to this issue

Most upvoted comments

I believe the KEYS command should be avoided entirely. It’s comparable to Rails performing a full sequential scan (without an index) across an entire relational database. While it might work fine in a local environment or small production setups, it would be a disaster in real-world systems. With Redis, the situation is even worse because the command blocks the entire server.

Scanning Redis once (e.g., during startup) is also problematic, as it still blocks the server for seconds and could prevent other clients or gems from initializing. In the best-case scenario, replacing KEYS with batched SCAN could result in Sidekiq being delayed by ~10–inf seconds, depending on the size of the Redis database.

Option 2 aligns with how Sidekiq’s queues work, and everything I’ve mentioned explains why it’s implemented this way.

I’ve looked into it, and it seems that using SCAN instead of KEYS offers limited benefit. While SCAN prevents Sidekiq from blocking the entire Redis server for [many] seconds, each call to Sidekiq::Cron::Namespace.{all,all_with_count} would still take considerable time (basically time for KEYS plus extra latency overhead), making it practically unusable.

Note that Sidekiq itself explicitly avoids using this type of query (O(n), where n is total keys count) by working with queues specified at configuration time. Iterating over jobs within a specific queue only requires a single lookup in Redis’s queue structure.

ATM we will probably just monkeypatch this method like this

class Sidekiq::Cron::Namespace
  def self.all
    [Sidekiq::Cron.configuration.default_namespace]
  end
end

but it would be cool to make the entire “namespacing” optional.

There are two steps we can take to mitigate the issue. First, while I completely understand that the namespaces = conn.keys('cron_jobs:*').collect do |key| part was introduced in version 2.0.0 as an implementation detail, it has a notable side effect of blocking the entire Redis server for a duration proportional to the number of keys. I believe we are not the only Sidekiq-Cron users with a database of this size (which I wouldn’t consider “extremely huge”, just relatively big), so I think it should be mentioned that the gem now performs a full database scan at startup (maybe in release description?). The issue is hard (if not impossible) to catch in staging environments for obvious reasons.

Secondly, it might be possible to replace the KEYS command with SCAN, which would alleviate some of the blocking issues. I’m not familiar with the codebase, so I’m not yet sure if this approach is applicable here, but I’ll take a look.

@taylorrf We were running version 2.0.1. The KEYS command was blocking Redis instance and some of requests began timing out. With config update and 2.2.0 you should be fine.

@viralpraxis I have not dug into the details of the namespaces feature, but my understanding is that if it is not explicitly configured it will fallback to the default namespace. The same default namespace is configured with available_namespaces and nothing will break. Or will it?

Either way, this is still just a workaround and the main fix should be not to use the keys command at all. From my perspective, namespaces can’t be used now in high-load environments without changin that.

Even official Redis documentation says that keys should be never used in production.

Hi, I’ve just had an issue in production triggered by this keys call.

I’d like to suggest a potential improvement:

  1. Disabling namespaces by default: This would eliminate the KEYS command usage for most users who don’t explicitly need namespacing
  2. Using SCAN instead of KEYS: This is a cursor-based approach that avoids blocking the Redis server

This change would significantly improve the performance and scalability of Sidekiq Cron, particularly for users with extensive Redis deployments.

Hello. Just wanted to +1 and mention that my company is also seeing this issue. We have a dozens of ruby services which use sidekiq-cron gem and keys command is also forbidden by our database ops team for a good reason - as was already mentioned above. So we can’t upgrade to the versions 2 and above.

Thank you so much for working on the fix!

If we take this path (moving it to configuration option), I think we should keep anyway the namespacing feature always on, by using the “default” as the default available namespace.

It also remains me how the I18n module works and its available_locales setting.

Maybe it’s the only way to avoid the whole scan? This approach (available_locales with fallback to full scan) seems probably fine https://github.com/sidekiq-cron/sidekiq-cron/issues/516#issuecomment-2482731532.

I don’t really see how we can fix the things and keep total backwards compatibility. Given that we all agree that performing KEYS by default is unacceptable here’s the best way to manage this issue I can think of:

  1. Change default value of available_namespaces from nil to %w[default]
  2. Add support for special value available_namespaces = :auto which will infer available namespaces using KEYS

It will be backward-incompatible only for people using namespacing and not specifying them explicitly – they’ll have to either specify :auto (to keep current behaviour) or list all the used namespaces by hand; new users and users not using namespacing will be fine.

Honestly, I agree with you that namespacing should be optional and disabled by default. We can achieve this simply by setting available_namespaces to %w[default] instead of nil on the gem’s side. The main issue right now is that it’s not entirely clear to users that using sidekiq-cron without explicitly configured namespaces can break apps with large Redis deployments—and that’s very wrong, in my opinion.

Hey @eileen-jiang, that’s correct.

Can we just emit a warning and assign the “default” namespace?

That sounds good to me.

So to recap, this gem need a available_namespaces configuration option that:

  • when is not defined would keep the current behavior (meaning using the Redis KEYS method)
  • when defined would be used to return the list of namespaces AND warn when a job is created with a namespace which is not included in that list, but use the default namespace

The third option is to retain the current behavior if available_namespaces is not specified (or e.g. set to special value :automatic). This approach would satisfy both users who want it to work automatically and those of us managing large Redis deployments.

Yes, I think we have 2 options:

  • New structure to save available “namespaces” in Redis (as @zedtux pointed out above).
  • Make it like a configuration option, ala available_namespaces.

Not sure which one is better 😅

I can confirm that KEYS cron_jobs:* is performed each time /sidekiq/cron is accessed.