rippled: rippled v1.3.1 OOM (out of memory) crash on testnet

Hello,every one ,I know the reason for my question, may be the same reason for yours @gituser .

The g_libc will hold the memory instead of free to OS after the destructor is called , you can use malloc_trim(0) to free the memory g_libc not freed,see :https://stackoverflow.com/questions/15529643/what-does-malloc-trim0-really-mean

I have a similiar problem when using rippled:

There is a project using rippled0.80.2, and it have ran abount a year,with 180G data(sqlite+rocksdb)， Environment: aws c5d.2xlarge 8cpu 16G NVMe SSD and the sles count are below:

 {
         "account" : 2524069,
         "amendments" : 1,
         "directory" : 2518953,
         "escrow" : 0,
         "fee" : 0,
         "hashes" : 136,
         "offer" : 1,
         "payment_channel" : 0,
         "signer_list" : 0,
         "state" : 4738656,
         "table" : 0,
         "ticket" : 0
 }

I am puzzled about the memory in rippled process. The key point is when the treenodecache size returned from ‘get_counts’ get down from above 12000000 to below 2000000, and other params didn’t grow, but the memory didn’t get down at all(Always about 10G). So I suspect there is a memory leak in rippled.

The json returned by get_counts the first time:

{
   "id" : 1,
   "result" : {
      "AL_hit_rate" : 15.70247936248779,
      "HashRouterEntry" : 1479,
      "InboundLedger" : 99,
      "Ledger" : 109,
      "NodeObject" : 17812,
      "RCLCxPeerPos::Data" : 40,
      "SLE_hit_rate" : 0.3076923076923077,
      "STArray" : 128,
      "STLedgerEntry" : 29,
      "STObject" : 1332,
      "STTx" : 244,
      "STValidation" : 392,
      "Transaction" : 173,
      "dbKBLedger" : 4204,
      "dbKBTotal" : 9604,
      "dbKBTransaction" : 4204,
      "fullbelow_size" : 2559678,
      "historical_perminute" : 22,
      "ledger_hit_rate" : 11.01928329467773,
      "node_hit_rate" : 28.22913932800293,
      "node_read_bytes" : 4163320653,
      "node_reads_hit" : 14754404,
      "node_reads_total" : 15643782,
      "node_writes" : 741054,
      "node_written_bytes" : 289595474,
      "status" : "success",
      "treenode_cache_size" : 12238,
      "treenode_track_size" : 12009843,
      "uptime" : "5 minutes, 5 seconds",
      "write_load" : 0
   }
}

The second:

{
   "id" : 1,
   "result" : {
      "AL_hit_rate" : 30.69767570495605,
      "HashRouterEntry" : 1509,
      "Ledger" : 14,
      "NodeObject" : 8539,
      "RCLCxPeerPos::Data" : 40,
      "SLE_hit_rate" : 0.5398058252427185,
      "STArray" : 102,
      "STLedgerEntry" : 141,
      "STObject" : 1301,
      "STTx" : 297,
      "STValidation" : 584,
      "Transaction" : 274,
      "dbKBLedger" : 4204,
      "dbKBTotal" : 9604,
      "dbKBTransaction" : 4204,
      "fullbelow_size" : 1374097,
      "historical_perminute" : 0,
      "ledger_hit_rate" : 15.36214923858643,
      "node_hit_rate" : 28.27434730529785,
      "node_read_bytes" : 4203252193,
      "node_reads_hit" : 14893786,
      "node_reads_total" : 15783684,
      "node_writes" : 743631,
      "node_written_bytes" : 291259310,
      "status" : "success",
      "treenode_cache_size" : 163,
      "treenode_track_size" : 1625090,
      "uptime" : "7 minutes, 38 seconds",
      "write_load" : 0
   }
}

It could very well be that your node doesn’t manage to write its files fast enough and fills caches until it runs out of memory. NuDB caches much less (and also does not rewrite/compact data into different levels), so that’s one way you could try to find out what happens.

Other cryptocurrency clients also have their issues (I ran into some parity based servers that are hilariously bad at dealing with RocksDB), many don’t have as much state or write/read load as XRPL so they can cache more or can take their time to do a compaction since the next block is still minutes away.

I’m not saying there isn’t an issue within rippled potentially, it’s just that a resource starved server (too little network IO, disk IO, CPU or RAM) tends to spiral out of control a bit in my experience. I had other issues with a spotty and slow internet connection, which lead to rippled constantly fully re-syncing which led to even more traffic. It might very well be the case that on your node it tries to write to the database but can’t because it is compacting or still slowly writing so it will cache and cache and cache…

Running rippled without a fast SSD is anyways a futile task if you want any kind of historic data. Why do you need exactly 2000 ledgers anyways? Maybe you can increase/decrease that number to see if it helps. Also you could start logging below the “warning” level and also monitoring the machine so you have a bit more insight into what’s happening.