ardb: ARDB segmentation fault for >37GB of data

I am trying to stress test ARDB. It seems after using redis-benchmark heavily ARDB dies

redis-benchmark -p 22122 -c 10 -d 1024 -n 500000000 -t set -r 500000000

However, I cannot find out the core dump to debug. I am assuming there is some leak. Any advice would be highly appreciated. The log file are not informative about an ERROR

[11256] 03-18 02:00:12,611 WARN [RocksDB]------- DUMPING STATS -------
[11256] 03-18 02:00:12,611 WARN [RocksDB]
** Compaction Stats [default] **
Level    Files   Size(MB} Score Read(GB}  Rn(GB} Rnp1(GB} Write(GB} Wnew(GB} Moved(GB} W-Amp Rd(MB/s} Wr(MB/s} Comp(sec} Comp(cnt} Avg(sec} KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0       0.00   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
 Int      0/0       0.00   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0         0         0    0.000       0      0
Uptime(secs): 19130.5 total, 19130.5 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interva
[11256] 03-18 02:00:27,189 WARN [RocksDB][0] Increasing compaction threads because we have 5 level-0 files 
[11256] 03-18 02:01:13,765 WARN [RocksDB][0] Increasing compaction threads because we have 5 level-0 files 
[11256] 03-18 02:01:20,712 WARN [RocksDB][0] Increasing compaction threads because we have 5 level-0 files 
[11256] 03-18 02:01:34,180 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:01:37,628 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:01:37,700 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:01:48,507 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:01:49,264 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:12,577 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:16,869 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:17,000 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:23,320 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:26,122 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:28,721 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:30,964 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:02:31,624 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:03:11,284 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:03:13,304 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:03:18,078 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:03:19,252 WARN [RocksDB][0] Increasing compaction threads because we have 6 level-0 files 
[11256] 03-18 02:03:23,529 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:03:56,490 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:04:09,855 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:04:13,452 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:04:23,760 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:04:30,957 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:04:36,614 WARN [RocksDB][0] Increasing compaction threads because we have 7 level-0 files 
[11256] 03-18 02:04:45,029 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files 
[11256] 03-18 02:05:09,373 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files 
[11256] 03-18 02:05:16,277 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files 
[11256] 03-18 02:05:24,036 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files 
[11256] 03-18 02:05:29,957 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files 
[11256] 03-18 02:05:39,676 WARN [RocksDB][0] Increasing compaction threads because we have 8 level-0 files 
[11256] 03-18 02:05:46,104 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files 
[11256] 03-18 02:05:52,865 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files 
[11256] 03-18 02:06:05,765 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files 
[11256] 03-18 02:06:30,504 WARN [RocksDB][0] Increasing compaction threads because we have 9 level-0 files 
[11256] 03-18 02:06:36,652 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files 
[11256] 03-18 02:06:39,713 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files 
[11256] 03-18 02:06:53,097 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files 
[11256] 03-18 02:07:13,035 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files 
[11256] 03-18 02:07:16,876 WARN [RocksDB][0] Increasing compaction threads because we have 10 level-0 files 
[11256] 03-18 02:07:26,102 WARN [RocksDB][0] Increasing compaction threads because we have 11 level-0 files 
[11256] 03-18 02:07:45,854 WARN [RocksDB][0] Increasing compaction threads because we have 11 level-0 files 
[11256] 03-18 02:08:16,624 WARN [RocksDB][0] Increasing compaction threads because we have 11 level-0 files 
[11256] 03-18 02:08:25,522 WARN [RocksDB][0] Increasing compaction threads because we have 12 level-0 files 
[11256] 03-18 02:08:46,459 WARN [RocksDB][0] Increasing compaction threads because we have 12 level-0 files 
[11256] 03-18 02:09:18,563 WARN [RocksDB][0] Increasing compaction threads because we have 12 level-0 files 
[11256] 03-18 02:09:27,853 WARN [RocksDB][0] Increasing compaction threads because we have 13 level-0 files 
[11256] 03-18 02:09:49,285 WARN [RocksDB][0] Increasing compaction threads because we have 13 level-0 files 
[11256] 03-18 02:10:02,817 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24819022457
[11256] 03-18 02:10:07,747 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24573715630
[11256] 03-18 02:10:07,873 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24327613024
[11256] 03-18 02:10:07,890 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 24080692714
[11256] 03-18 02:10:08,053 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 23832934071
[11256] 03-18 02:10:08,132 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 23584314769
[11256] 03-18 02:10:12,793 WARN [RocksDB][0] Increasing compaction threads because of estimated pending compaction bytes 23334718288
[11256] 03-18 02:10:12,793 WARN [RocksDB]------- DUMPING STATS -------

I am using /tmp for the code dumps but the folder does not have any core dumps.

/tmp $ cat /proc/sys/kernel/core_pattern 
/tmp/core_%e.%p

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Comments: 16 (11 by maintainers)

Commits related to this issue

Most upvoted comments

I’m using ardb with rocksdb. Currently my DB size is 37GB, ~78M keys. No crashes so far. Here is my image: https://hub.docker.com/r/expert/ardb/