KeyDB: [BUG] Flash key expire routine -- 100% CPU
Describe the bug
With flash enabled on v6.3.4, I wrote a script that loads up the DB with > 3mil keys and majority having TTL expiring in 24hrs. Comes the next day, server.cpp will call the expire routine:
activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW);
When this happens, I observed the CPU going up to 100%. Below is the stats printed out to logs, showing keys being expired (roughly 200 - 400 every 10s).
41302:41354:M 27 Nov 2023 08:48:33.788 # DB 0: 3108939 keys (2524664 volatile) in 2097152 slots HT.
41302:41354:M 27 Nov 2023 08:48:43.707 # DB 0: 3108684 keys (2524409 volatile) in 2097152 slots HT.
41302:41354:M 27 Nov 2023 08:48:53.372 # DB 0: 3108434 keys (2524159 volatile) in 2097152 slots HT.
41302:41354:M 27 Nov 2023 08:49:03.004 # DB 0: 3108183 keys (2523908 volatile) in 2097152 slots HT.
41302:41354:M 27 Nov 2023 08:49:13.252 # DB 0: 3107926 keys (2523651 volatile) in 2097152 slots HT.
41302:41354:M 27 Nov 2023 08:49:23.298 # DB 0: 3107677 keys (2523402 volatile) in 2097152 slots HT.
Running flamegraph, I can see KeyDB is getting into spin lock during the expire routine above. Is there anyway I can bring the CPU usage back down to a reasonable value so that clients become more responsive?
Expected behavior
CPU should be kept moderately low during expire routine.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 17
@gloomy-7164, just referencing this here, just in case you might find it useful during your testing:
https://github.com/Snapchat/KeyDB/issues/749
It contains the stuck issue you discovered, as well as the scenarios in which the expiry processing routine should handle.
First issue with your idea: If you end up caching the last key, then the function will not work anymore, it will seek to the incorrect place each time it is called. Eg you cache 1:00pm, and a new key inserted with TTL at 9am, the next call it will seek to 1:00pm, missing the new key with earlier TTL.
Second issue. When there is no expiry keys (eg all keys eventually expired), the function will end up calling SeekToFirst each time, causing CPU to spike up again. So, you need to keep track of when the DB is empty, eg using a isEmpty flag. Only when isEmpty is false, you continue, else do an early exit.
Seek is in the microseconds, if you provide the correct key to seek. But to achieve this, you have to do a lot of work. One more thing you can also look at. Initially, I tried to reduce the CPU load of getExpiryCandidates() using other methods, ie only doing 1 run per loop in activeExpireCycleCore, or returning less records in getExpiryCandidates. They didn’t achieve the desired results, only the change from SeekToFirst to Seek made it back to 2% CPU. But I still haven’t removed these earlier hacks, so CPU might be higher if I reverted them.
Eviction is only done when it needs to free space, but expiry is done at a constant higher frequency. So, the CPU impact is not high enough to cause any issues (so far). But there are other eviction issues (not related to flash) which you can search in the issues.