KeyDB: [CRASH] ckeysExpired == db->expireSize()
Crash report
407496:397491:C 19 Nov 2023 11:01:19.885 * RDB: 4 MB of memory used by copy-on-write
397462:397491:M 19 Nov 2023 11:01:19.992 * Background saving terminated with success
397462:397491:M 19 Nov 2023 11:11:59.701 * 10 changes in 300 seconds. Saving...
397462:397491:M 19 Nov 2023 11:11:59.773 * Background saving started by pid 407927
397462:397491:M 19 Nov 2023 11:11:59.773 * Background saving started
407927:397491:C 19 Nov 2023 11:12:13.422 * DB saved on disk
407927:397491:C 19 Nov 2023 11:12:13.476 * RDB: 6 MB of memory used by copy-on-write
397462:397491:M 19 Nov 2023 11:12:13.626 * Background saving terminated with success
397462:397491:M 19 Nov 2023 11:17:14.080 * 10 changes in 300 seconds. Saving...
397462:397491:M 19 Nov 2023 11:17:14.166 * Background saving started by pid 408145
397462:397491:M 19 Nov 2023 11:17:14.166 * Background saving started
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
408145:397491:C 19 Nov 2023 11:17:28.270 # === ASSERTION FAILED ===
408145:397491:C 19 Nov 2023 11:17:28.270 # ==> rdb.cpp:1372 'ckeysExpired == db->expireSize()' is not true
------ STACK TRACE ------
Backtrace:
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x58e) [0x5569ce3f740e]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x115) [0x5569ce3f85d5]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x73) [0x5569ce3f8963]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackgroundFork(rdbSaveInfo*)+0x122) [0x5569ce3f91d2]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackground(rdbSaveInfo*)+0xd8) [0x5569ce3f9888]
keydb-rdb-bgsave 192.168.254.52:6379(serverCron(aeEventLoop*, long long, void*)+0xe14) [0x5569ce39c854]
keydb-rdb-bgsave 192.168.254.52:6379(aeProcessEvents+0x224) [0x5569ce3934a4]
keydb-rdb-bgsave 192.168.254.52:6379(aeMain+0x3e) [0x5569ce393a6e]
keydb-rdb-bgsave 192.168.254.52:6379(workerThreadMain(void*)+0xfe) [0x5569ce3abe9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f24b3125ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a40) [0x7f24b31b7a40]
------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b532cd0401cb0da4
redis_mode:standalone
os:Linux 5.15.0-88-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:408145
process_supervised:no
run_id:2ee93088a878f55d0bdc926eb51adf235d1d7439
tcp_port:6379
server_time_usec:1700392634080800
uptime_in_seconds:16315
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5893818
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb-6379.conf
availability_zone:
features:cluster_mget
# Clients
connected_clients:55
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:20
thread_1_clients:21
thread_2_clients:14
thread_3_clients:0
# Memory
used_memory:2211691160
used_memory_human:2.06G
used_memory_rss:2272120832
used_memory_rss_human:2.12G
used_memory_peak:14828083264
used_memory_peak_human:13.81G
used_memory_peak_perc:14.92%
used_memory_overhead:28263272
used_memory_startup:3287000
used_memory_dataset:2183427888
used_memory_dataset_perc:98.87%
allocator_allocated:2213289064
allocator_active:2225258496
allocator_resident:2321199104
total_system_memory:134980608000
total_system_memory_human:125.71G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:51539607552
maxmemory_human:48.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.01
allocator_frag_bytes:11969432
allocator_rss_ratio:1.04
allocator_rss_bytes:95940608
rss_overhead_ratio:0.98
rss_overhead_bytes:-49078272
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:60431456
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1127776
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:9193
rdb_bgsave_in_progress:0
rdb_last_save_time:1700392333
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:14
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:6316032
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:88
total_commands_processed:9319761
instantaneous_ops_per_sec:0
total_net_input_bytes:106185379481
total_net_output_bytes:798164883
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:752200
expired_stale_perc:0.00
expired_time_cap_reached_count:5
expire_cycle_cpu_milliseconds:88799
evicted_keys:0
keyspace_hits:11211
keyspace_misses:2444
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:72330
total_forks:21
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:18951592
total_writes_processed:9319802
instantaneous_lock_contention:1
avg_lock_contention:0.031250
storage_provider_read_hits:0
storage_provider_read_misses:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:bfd56f64a0d362477baa6f6689c00df36bad0667
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:1.409685
used_cpu_user:12.014361
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:108
used_cpu_sys_main_thread:1.409685
used_cpu_user_main_thread:12.014361
# Modules
# Commandstats
cmdstat_get:calls=13655,usec=193595,usec_per_call=14.18,rejected_calls=0,failed_calls=0
cmdstat_del:calls=2880411,usec=13713807,usec_per_call=4.76,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=6,usec=3,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=2886,usec=31633,usec_per_call=10.96,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=88,usec=281,usec_per_call=3.19,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=2,usec=3,usec_per_call=1.50,rejected_calls=0,failed_calls=0
cmdstat_set:calls=3211387,usec=8627542,usec_per_call=2.69,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3211326,usec=5992111,usec_per_call=1.87,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=322081,expires=2343,avg_ttl=2386193452,cached_keys=322081
# KeyDB
mvcc_depth:0
------ CLIENT LIST OUTPUT ------
id=48 addr=192.168.254.53:33122 laddr=192.168.254.52:6379 fd=37 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=74 addr=192.168.254.52:48314 laddr=192.168.254.52:6379 fd=63 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=75 addr=192.168.254.53:33248 laddr=192.168.254.52:6379 fd=64 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=76 addr=192.168.254.52:48390 laddr=192.168.254.52:6379 fd=69 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=49 addr=192.168.254.53:33134 laddr=192.168.254.52:6379 fd=38 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=50 addr=192.168.254.52:48262 laddr=192.168.254.52:6379 fd=40 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=51 addr=192.168.254.52:48260 laddr=192.168.254.52:6379 fd=39 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=72 addr=192.168.254.53:33246 laddr=192.168.254.52:6379 fd=61 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=73 addr=192.168.254.53:33234 laddr=192.168.254.52:6379 fd=62 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=52 addr=192.168.254.52:48266 laddr=192.168.254.52:6379 fd=41 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=53 addr=192.168.254.53:33144 laddr=192.168.254.52:6379 fd=43 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=54 addr=192.168.254.53:33150 laddr=192.168.254.52:6379 fd=42 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=55 addr=192.168.254.52:48274 laddr=192.168.254.52:6379 fd=44 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=71 addr=192.168.254.53:33228 laddr=192.168.254.52:6379 fd=60 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=56 addr=192.168.254.52:48278 laddr=192.168.254.52:6379 fd=45 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=57 addr=192.168.254.52:48276 laddr=192.168.254.52:6379 fd=46 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=58 addr=192.168.254.53:33158 laddr=192.168.254.52:6379 fd=47 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=59 addr=192.168.254.53:33166 laddr=192.168.254.52:6379 fd=48 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=92 addr=192.168.254.53:60818 laddr=192.168.254.52:6379 fd=81 name= age=5082 idle=315 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=60 addr=192.168.254.52:48280 laddr=192.168.254.52:6379 fd=49 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=61 addr=192.168.254.52:48286 laddr=192.168.254.52:6379 fd=50 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=62 addr=192.168.254.52:48298 laddr=192.168.254.52:6379 fd=51 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=63 addr=192.168.254.53:33172 laddr=192.168.254.52:6379 fd=53 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=64 addr=192.168.254.52:48306 laddr=192.168.254.52:6379 fd=54 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=65 addr=192.168.254.53:33174 laddr=192.168.254.52:6379 fd=52 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=79 addr=192.168.254.52:48406 laddr=192.168.254.52:6379 fd=70 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=80 addr=192.168.254.52:48356 laddr=192.168.254.52:6379 fd=71 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=81 addr=192.168.254.52:48368 laddr=192.168.254.52:6379 fd=72 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=82 addr=192.168.254.52:48412 laddr=192.168.254.52:6379 fd=73 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=83 addr=192.168.254.53:33294 laddr=192.168.254.52:6379 fd=74 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=84 addr=192.168.254.52:48378 laddr=192.168.254.52:6379 fd=75 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=85 addr=192.168.254.53:33256 laddr=192.168.254.52:6379 fd=76 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=86 addr=192.168.254.53:33272 laddr=192.168.254.52:6379 fd=77 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=87 addr=192.168.254.53:33284 laddr=192.168.254.52:6379 fd=78 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=88 addr=192.168.254.52:48336 laddr=192.168.254.52:6379 fd=65 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=89 addr=192.168.254.52:48320 laddr=192.168.254.52:6379 fd=66 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=90 addr=192.168.254.52:48366 laddr=192.168.254.52:6379 fd=67 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=91 addr=192.168.254.52:48346 laddr=192.168.254.52:6379 fd=68 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=66 addr=192.168.254.53:33186 laddr=192.168.254.52:6379 fd=55 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=67 addr=192.168.254.53:33196 laddr=192.168.254.52:6379 fd=56 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=68 addr=192.168.254.53:33206 laddr=192.168.254.52:6379 fd=57 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=77 addr=192.168.254.52:48404 laddr=192.168.254.52:6379 fd=79 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=78 addr=192.168.254.53:33298 laddr=192.168.254.52:6379 fd=80 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=69 addr=192.168.254.53:33212 laddr=192.168.254.52:6379 fd=58 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=70 addr=192.168.254.53:33214 laddr=192.168.254.52:6379 fd=59 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=39 addr=192.168.254.53:38410 laddr=192.168.254.52:6379 fd=28 name= age=9493 idle=968 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=40 addr=192.168.254.53:38424 laddr=192.168.254.52:6379 fd=29 name= age=9493 idle=223 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=41 addr=192.168.254.53:38430 laddr=192.168.254.52:6379 fd=32 name= age=9492 idle=142 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=42 addr=192.168.254.53:47514 laddr=192.168.254.52:6379 fd=30 name= age=8569 idle=66 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=93 addr=192.168.254.53:60450 laddr=192.168.254.52:6379 fd=82 name= age=848 idle=848 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=43 addr=192.168.254.52:55578 laddr=192.168.254.52:6379 fd=34 name= age=6259 idle=6259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=44 addr=192.168.254.53:33090 laddr=192.168.254.52:6379 fd=31 name= age=6173 idle=141 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=45 addr=192.168.254.52:48246 laddr=192.168.254.52:6379 fd=33 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=46 addr=192.168.254.53:33106 laddr=192.168.254.52:6379 fd=35 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=47 addr=192.168.254.52:48254 laddr=192.168.254.52:6379 fd=36 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
408145:397491:C 19 Nov 2023 11:17:28.272 # Bio thread for job type #0 terminated
408145:397491:C 19 Nov 2023 11:17:28.272 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
408145:397491:C 19 Nov 2023 11:17:28.272 # Accessing address: 0x7f24b0e95910
408145:397491:C 19 Nov 2023 11:17:28.272 # Crashed running the instruction at: 0x7f24b3123bdd
------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(pthread_cancel+0x1d) [0x7f24b3123bdd]
Backtrace:
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f24b30d3520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cancel+0x1d) [0x7f24b3123bdd]
keydb-rdb-bgsave 192.168.254.52:6379(bioKillThreads+0x65) [0x5569ce45c825]
keydb-rdb-bgsave 192.168.254.52:6379(doFastMemoryTest()+0x3f) [0x5569ce43860f]
keydb-rdb-bgsave 192.168.254.52:6379(_serverAssert+0xc6) [0x5569ce438746]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveRio(_rio*, redisDbPersistentDataSnapshot const**, int*, int, rdbSaveInfo*)+0x58e) [0x5569ce3f740e]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveFile(char*, redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x115) [0x5569ce3f85d5]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSave(redisDbPersistentDataSnapshot const**, rdbSaveInfo*)+0x73) [0x5569ce3f8963]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackgroundFork(rdbSaveInfo*)+0x122) [0x5569ce3f91d2]
keydb-rdb-bgsave 192.168.254.52:6379(rdbSaveBackground(rdbSaveInfo*)+0xd8) [0x5569ce3f9888]
keydb-rdb-bgsave 192.168.254.52:6379(serverCron(aeEventLoop*, long long, void*)+0xe14) [0x5569ce39c854]
keydb-rdb-bgsave 192.168.254.52:6379(aeProcessEvents+0x224) [0x5569ce3934a4]
keydb-rdb-bgsave 192.168.254.52:6379(aeMain+0x3e) [0x5569ce393a6e]
keydb-rdb-bgsave 192.168.254.52:6379(workerThreadMain(void*)+0xfe) [0x5569ce3abe9e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f24b3125ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126a40) [0x7f24b31b7a40]
------ REGISTERS ------
408145:397491:C 19 Nov 2023 11:17:28.273 #
RAX:b4973fcc4d1b9400 RBX:0000000000000001
RCX:0000000000000002 RDX:ffffffffffffe308
RDI:00007f24b0e95640 RSI:00007f242c9fcb10
RBP:00007f242c9fe640 RSP:00007f242c9fa3b0
R8 :00005569cec93c50 R9 :0000000000000200
R10:00007f24b2200900 R11:0000000000000010
R12:00005569cea88a30 R13:00005569ce921bb0
R14:00005569ce921be8 R15:000000000004ea21
RIP:00007f24b3123bdd EFL:0000000000010202
CSGSFS:002b000000000033
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bf) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3be) -> 00007f24ad9ffc60
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bd) -> b4973fcc4d1b9400
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bc) -> 00007f242a20508b
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3bb) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3ba) -> 0000000600000007
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b9) -> 00007f24b1696c68
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b8) -> 00007f24b31274d0
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b7) -> 00007f242c9fa3f0
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b6) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b5) -> 00007f24b312759c
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b4) -> 0000000000000000
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b3) -> 00007f24b1696640
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b2) -> 00007f24b3123b30
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b1) -> 00000000000001c0
408145:397491:C 19 Nov 2023 11:17:28.273 # (00007f242c9fa3b0) -> 00005569cec93c50
------ INFO OUTPUT ------
# Server
redis_version:6.3.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b532cd0401cb0da4
redis_mode:standalone
os:Linux 5.15.0-88-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:408145
process_supervised:no
run_id:2ee93088a878f55d0bdc926eb51adf235d1d7439
tcp_port:6379
server_time_usec:1700392634080800
uptime_in_seconds:16315
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5893818
executable:/usr/bin/keydb-server
config_file:/etc/keydb/keydb-6379.conf
availability_zone:
features:cluster_mget
# Clients
connected_clients:55
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:20
thread_1_clients:21
thread_2_clients:14
thread_3_clients:0
# Memory
used_memory:2211693800
used_memory_human:2.06G
used_memory_rss:2272120832
used_memory_rss_human:2.12G
used_memory_peak:14828083264
used_memory_peak_human:13.81G
used_memory_peak_perc:14.92%
used_memory_overhead:28263272
used_memory_startup:3287000
used_memory_dataset:2183430528
used_memory_dataset_perc:98.87%
allocator_allocated:2213289064
allocator_active:2225258496
allocator_resident:2321199104
total_system_memory:134980608000
total_system_memory_human:125.71G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:51539607552
maxmemory_human:48.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.01
allocator_frag_bytes:11969432
allocator_rss_ratio:1.04
allocator_rss_bytes:95940608
rss_overhead_ratio:0.98
rss_overhead_bytes:-49078272
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:60431456
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1127776
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:9193
rdb_bgsave_in_progress:0
rdb_last_save_time:1700392333
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:14
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:6316032
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:88
total_commands_processed:9319761
instantaneous_ops_per_sec:0
total_net_input_bytes:106185379481
total_net_output_bytes:798164883
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:752200
expired_stale_perc:0.00
expired_time_cap_reached_count:5
expire_cycle_cpu_milliseconds:88799
evicted_keys:0
keyspace_hits:11211
keyspace_misses:2444
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:72330
total_forks:21
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:18951592
total_writes_processed:9319802
instantaneous_lock_contention:1
avg_lock_contention:0.031250
storage_provider_read_hits:0
storage_provider_read_misses:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:bfd56f64a0d362477baa6f6689c00df36bad0667
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:1.409841
used_cpu_user:12.015691
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:108
used_cpu_sys_main_thread:1.409841
used_cpu_user_main_thread:12.015691
# Modules
# Commandstats
cmdstat_get:calls=13655,usec=193595,usec_per_call=14.18,rejected_calls=0,failed_calls=0
cmdstat_del:calls=2880411,usec=13713807,usec_per_call=4.76,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=6,usec=3,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=2886,usec=31633,usec_per_call=10.96,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=88,usec=281,usec_per_call=3.19,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=2,usec=3,usec_per_call=1.50,rejected_calls=0,failed_calls=0
cmdstat_set:calls=3211387,usec=8627542,usec_per_call=2.69,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3211326,usec=5992111,usec_per_call=1.87,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=322081,expires=2343,avg_ttl=2386193452,cached_keys=322081
# KeyDB
mvcc_depth:0
------ CLIENT LIST OUTPUT ------
id=48 addr=192.168.254.53:33122 laddr=192.168.254.52:6379 fd=37 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=74 addr=192.168.254.52:48314 laddr=192.168.254.52:6379 fd=63 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=75 addr=192.168.254.53:33248 laddr=192.168.254.52:6379 fd=64 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=76 addr=192.168.254.52:48390 laddr=192.168.254.52:6379 fd=69 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=49 addr=192.168.254.53:33134 laddr=192.168.254.52:6379 fd=38 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=50 addr=192.168.254.52:48262 laddr=192.168.254.52:6379 fd=40 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=51 addr=192.168.254.52:48260 laddr=192.168.254.52:6379 fd=39 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=72 addr=192.168.254.53:33246 laddr=192.168.254.52:6379 fd=61 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=73 addr=192.168.254.53:33234 laddr=192.168.254.52:6379 fd=62 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=52 addr=192.168.254.52:48266 laddr=192.168.254.52:6379 fd=41 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=53 addr=192.168.254.53:33144 laddr=192.168.254.52:6379 fd=43 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=54 addr=192.168.254.53:33150 laddr=192.168.254.52:6379 fd=42 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=55 addr=192.168.254.52:48274 laddr=192.168.254.52:6379 fd=44 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=71 addr=192.168.254.53:33228 laddr=192.168.254.52:6379 fd=60 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=56 addr=192.168.254.52:48278 laddr=192.168.254.52:6379 fd=45 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=57 addr=192.168.254.52:48276 laddr=192.168.254.52:6379 fd=46 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=58 addr=192.168.254.53:33158 laddr=192.168.254.52:6379 fd=47 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=59 addr=192.168.254.53:33166 laddr=192.168.254.52:6379 fd=48 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=92 addr=192.168.254.53:60818 laddr=192.168.254.52:6379 fd=81 name= age=5082 idle=315 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=60 addr=192.168.254.52:48280 laddr=192.168.254.52:6379 fd=49 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=61 addr=192.168.254.52:48286 laddr=192.168.254.52:6379 fd=50 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=62 addr=192.168.254.52:48298 laddr=192.168.254.52:6379 fd=51 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=63 addr=192.168.254.53:33172 laddr=192.168.254.52:6379 fd=53 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=64 addr=192.168.254.52:48306 laddr=192.168.254.52:6379 fd=54 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=65 addr=192.168.254.53:33174 laddr=192.168.254.52:6379 fd=52 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=79 addr=192.168.254.52:48406 laddr=192.168.254.52:6379 fd=70 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=80 addr=192.168.254.52:48356 laddr=192.168.254.52:6379 fd=71 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=81 addr=192.168.254.52:48368 laddr=192.168.254.52:6379 fd=72 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=82 addr=192.168.254.52:48412 laddr=192.168.254.52:6379 fd=73 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=83 addr=192.168.254.53:33294 laddr=192.168.254.52:6379 fd=74 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=84 addr=192.168.254.52:48378 laddr=192.168.254.52:6379 fd=75 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=85 addr=192.168.254.53:33256 laddr=192.168.254.52:6379 fd=76 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=86 addr=192.168.254.53:33272 laddr=192.168.254.52:6379 fd=77 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=87 addr=192.168.254.53:33284 laddr=192.168.254.52:6379 fd=78 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=88 addr=192.168.254.52:48336 laddr=192.168.254.52:6379 fd=65 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=89 addr=192.168.254.52:48320 laddr=192.168.254.52:6379 fd=66 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=90 addr=192.168.254.52:48366 laddr=192.168.254.52:6379 fd=67 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=91 addr=192.168.254.52:48346 laddr=192.168.254.52:6379 fd=68 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=66 addr=192.168.254.53:33186 laddr=192.168.254.52:6379 fd=55 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=67 addr=192.168.254.53:33196 laddr=192.168.254.52:6379 fd=56 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=68 addr=192.168.254.53:33206 laddr=192.168.254.52:6379 fd=57 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=77 addr=192.168.254.52:48404 laddr=192.168.254.52:6379 fd=79 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=78 addr=192.168.254.53:33298 laddr=192.168.254.52:6379 fd=80 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=69 addr=192.168.254.53:33212 laddr=192.168.254.52:6379 fd=58 name= age=6173 idle=216 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=70 addr=192.168.254.53:33214 laddr=192.168.254.52:6379 fd=59 name= age=6173 idle=213 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=39 addr=192.168.254.53:38410 laddr=192.168.254.52:6379 fd=28 name= age=9493 idle=968 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=40 addr=192.168.254.53:38424 laddr=192.168.254.52:6379 fd=29 name= age=9493 idle=223 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=41 addr=192.168.254.53:38430 laddr=192.168.254.52:6379 fd=32 name= age=9492 idle=142 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=42 addr=192.168.254.53:47514 laddr=192.168.254.52:6379 fd=30 name= age=8569 idle=66 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=93 addr=192.168.254.53:60450 laddr=192.168.254.52:6379 fd=82 name= age=848 idle=848 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=43 addr=192.168.254.52:55578 laddr=192.168.254.52:6379 fd=34 name= age=6259 idle=6259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=set user=default redir=-1
id=44 addr=192.168.254.53:33090 laddr=192.168.254.52:6379 fd=31 name= age=6173 idle=141 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=45 addr=192.168.254.52:48246 laddr=192.168.254.52:6379 fd=33 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=46 addr=192.168.254.53:33106 laddr=192.168.254.52:6379 fd=35 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=47 addr=192.168.254.52:48254 laddr=192.168.254.52:6379 fd=36 name= age=6173 idle=217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
397462:397491:M 19 Nov 2023 11:17:28.507 # Background saving cancelled
397462:397491:M 19 Nov 2023 11:17:28.607 * 10 changes in 300 seconds. Saving...
397462:397491:M 19 Nov 2023 11:17:28.696 * Background saving started by pid 408160
397462:397491:M 19 Nov 2023 11:17:28.696 * Background saving started
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
408160:397491:C 19 Nov 2023 11:17:42.750 # === ASSERTION FAILED ===
408160:397491:C 19 Nov 2023 11:17:42.750 # ==> rdb.cpp:1372 'ckeysExpired == db->expireSize()' is not true
I’m having the prolem on my production server. I’m not using replication.
From what I can tell, the crashes are triggered by a client connection making a change to the DB and triggering a write of the RDB file. The crash appears to happen during the write of the DB file. This then happens every time a new change is made to the DB.
I can stop the crashes by flushing the DB.
Unfortunately, this is a production server and there’s a limit to what I can do.
I’ve flushed the DB and then put the server under significant load with the following script. I basically create a large number of keys with large values and random expire times, and then deletes them again. Some will naturally expire before I get round to deleting. This appears to work fine.
#!/usr/bin/env python
import random
import redis
import secrets
import threading
import time
REDIS_SERVER = dict(host='192.168.254.52', port=6379, db=0, password="*****")
def set_keys():
client = redis.Redis(**REDIS_SERVER)
v = secrets.token_hex(16384)
keys = []
for i in range(65536):
k = secrets.token_hex(32)
keys += [k]
client.set(k, v)
client.expire(k, random.randint(1, 32))
if len(keys) > 8192:
for k in keys:
client.delete(k)
keys = []
threads = []
for i in range(32):
thread = threading.Thread(target=set_keys)
threads.append(thread)
thread.start()
time.sleep(1)
for thread in threads:
thread.join()
With the system crashing, I’ve set up a second instance on an identical machine and used a python script to copy all keys, values and associated TTLs to the new instance. The second system works fine, dumps to the RDB when required.
If I point the production clients at this new instance after about 24 hours, the issue returns…so it’s not hardware related.
I have tried
enable-async-commands no
which I still have in place, but this does not resolve the issue.
It must be something to do with the contents of the DB or the way my clients (Python) are using it but I’m unable to isolate the problem.
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
About this issue
- Original URL
- State: open
- Created 7 months ago
- Reactions: 5
- Comments: 17
hi @rmd4
As per my previous comment, I tracked it down to the eviction changes for flash introduced in v6.3.4. My temporary workaround for this is to set m_numexpires on startup. So far, haven’t had the crash, ie with both flash and data dumping enabled.
For unclean shutdown, use a KILL -9 and also make sure you have entries with TTL in flash. And this may not be the only way to trigger the bug, as you have mentioned, it can also happen at runtime. I did find that once the variable is out of sync, then the crash can happen anytime (dump at shutdown or at runtime).
My current goal is to stress test the flash feature in the most challenging way I can, ie maxmemory smaller than dataset (so keys get evicted from memory), 5000+ read/writes per second, module scanning, RDB background saving, etc. I have had countless crashes, and just hacking the code to workaround them. So far, I am very impressed with KeyDB.