ardb: ARDB segmentation fault for >37GB of data
I am trying to stress test ARDB. It seems after using redis-benchmark heavily ARDB dies
redis-benchmark -p 22122 -c 10 -d 1024 -n 500000000 -t set -r 500000000
However, I cannot find out the core dump to debug. I am assuming there is some leak. Any advice would be highly appreciated. The log file are not informative about an ERROR
[11256] 03-18 02:00:12,611 WARN [RocksDB]------- DUMPING STATS -------
[11256] 03-18 02:00:12,611 WARN [RocksDB]
** Compaction Stats [default] **
Level Files Size(MB} Score Read(GB} Rn(GB} Rnp1(GB} Write(GB} Wnew(GB} Moved(GB} W-Amp Rd(MB/s} Wr(MB/s} Comp(sec} Comp(cnt} Avg(sec} KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
Sum 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Int 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
Uptime(secs): 19130.5 total, 19130.5 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interva
[11256] 03-18 02:00:27,189 WARN [RocksDB][0] Increasing compaction threads because we have 5 level-0 files
[11256] 03-18 02:01:13,765 WARN [RocksDB][0] Increasing compaction threads because we have 5 level-0 files
[11256] 03-18 02:01:20,712 WARN [RocksDB][0] Increasing compaction threads because we have 5 level-0 files
[11256] 03-18 02:01:34,180 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:01:37,628 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:01:37,700 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:01:48,507 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:01:49,264 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:12,577 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:16,869 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:17,000 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:23,320 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:26,122 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:28,721 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:30,964 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:02:31,624 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:03:11,284 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:03:13,304 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:03:18,078 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:03:19,252 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files
[11256] 03-18 02:03:23,529 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:03:56,490 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:04:09,855 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:04:13,452 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:04:23,760 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:04:30,957 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:04:36,614 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files
[11256] 03-18 02:04:45,029 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files
[11256] 03-18 02:05:09,373 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files
[11256] 03-18 02:05:16,277 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files
[11256] 03-18 02:05:24,036 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files
[11256] 03-18 02:05:29,957 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files
[11256] 03-18 02:05:39,676 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files
[11256] 03-18 02:05:46,104 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files
[11256] 03-18 02:05:52,865 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files
[11256] 03-18 02:06:05,765 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files
[11256] 03-18 02:06:30,504 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files
[11256] 03-18 02:06:36,652 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files
[11256] 03-18 02:06:39,713 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files
[11256] 03-18 02:06:53,097 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files
[11256] 03-18 02:07:13,035 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files
[11256] 03-18 02:07:16,876 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files
[11256] 03-18 02:07:26,102 WARN [RocksDB][0] Increasing compaction threads because we have 11 level-0 files
[11256] 03-18 02:07:45,854 WARN [RocksDB][0] Increasing compaction threads because we have 11 level-0 files
[11256] 03-18 02:08:16,624 WARN [RocksDB][0] Increasing compaction threads because we have 11 level-0 files
[11256] 03-18 02:08:25,522 WARN [RocksDB][0] Increasing compaction threads because we have 12 level-0 files
[11256] 03-18 02:08:46,459 WARN [RocksDB][0] Increasing compaction threads because we have 12 level-0 files
[11256] 03-18 02:09:18,563 WARN [RocksDB][0] Increasing compaction threads because we have 12 level-0 files
[11256] 03-18 02:09:27,853 WARN [RocksDB][0] Increasing compaction threads because we have 13 level-0 files
[11256] 03-18 02:09:49,285 WARN [RocksDB][0] Increasing compaction threads because we have 13 level-0 files
[11256] 03-18 02:10:02,817 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24819022457
[11256] 03-18 02:10:07,747 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24573715630
[11256] 03-18 02:10:07,873 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24327613024
[11256] 03-18 02:10:07,890 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24080692714
[11256] 03-18 02:10:08,053 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 23832934071
[11256] 03-18 02:10:08,132 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 23584314769
[11256] 03-18 02:10:12,793 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 23334718288
[11256] 03-18 02:10:12,793 WARN [RocksDB]------- DUMPING STATS -------
I am using /tmp
for the code dumps but the folder does not have any core dumps.
/tmp $ cat /proc/sys/kernel/core_pattern
/tmp/core_%e.%p
About this issue
- Original URL
- State: open
- Created 7 years ago
- Comments: 16 (11 by maintainers)
Commits related to this issue
- issue #327: fix for potential log crash — committed to yinqiwen/ardb by yinqiwen 7 years ago
I’m using ardb with rocksdb. Currently my DB size is 37GB, ~78M keys. No crashes so far. Here is my image: https://hub.docker.com/r/expert/ardb/