bitcoin: Slow memory leak in v22.0?

I’ve been noticing gradually increasing memory usage bitcoin core v22.0. I’ve been running this node for a couple of years with the same hardware and general setup and have only noticed this in the past few months, around the time I upgraded to v22, which makes me suspect it may be related.

I’m seeing bitcoind memory usage slowly climb over weeks/months of uptime. Right now it’s at 3.4 GB memory usage and as I recall I’ve seen it over 5 GB before in an incident where it caused the machine to run out of RAM, which is when I first noticed this. If I restart bitcoind, RAM usage goes back to a normal level and stays there for a while.

I do recall seeing a lot of BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications messages in the logs when RAM usage was over 5GB and my node was barely responsive to RPC calls, but I’m not sure if this has anything to do with the issue.

I’m using dbcache=1024 which would explain some of the RAM usage but not all of it.

Other config settings that may be relevant:

txindex=1
peerblockfilters=1
blockfilterindex=1

maxconnections=100

rpcthreads=64
rpcworkqueue=256
rpcservertimeout=120

zmqpubrawblockhwm=100000
zmqpubrawtxhwm=100000

Otherwise, I’m using this node to run an lnd node (with zmq block/tx notifications) and electrumx server. There’s a medium-sized watch-only wallet that’s loaded. There’s also a small script I run with blocknotify.

This is all running on an RPI4 with 8GB RAM, using raspbian buster w/ the OS and block indexes on an SSD.

I’d be interested if others have seen anything similar out of their nodes or if perhaps this RAM usage is normal given how I’m using it, but I am pretty sure this wasn’t happening a year ago with similar usage patterns.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 25 (23 by maintainers)

Commits related to this issue

Most upvoted comments

OK, I think I have pretty much identified the root cause here.

Calling blockToJSON on a maximum-sized block allocates 70-80MB on the heap during runtime.

There is then an additional 15-20MB allocated for strJSON.

After these allocations are made they then seem to be re-used when possible in the future, so long as subsequent blocks are small enough to fit. This totals up to the ~100MB RSS increase observed sometimes when getting blocks via REST.

In the event that blocks are fetched in series in increasing sizes, repeated allocations (of increasing sizes) may be made, causing even higher apparent resource usage.

The cause seems to be two-fold:

  1. UniValue’s are not a particularly compact representation (can’t do that much about this, except perhaps we could write to string directly for this method.)
  2. The UniValue created inside blockToJSON is ~40MB worst-case, but it is then copied into a second UniValue, doubling the required heap size.

It appears that this has been investgated previously (see also here) in search of better speed, but having a move semantic for UniValue would also shave off half of this function’s allocation requirement.

Thanks @stickies-v for helping me investigate

Looking at the logs, there are tons of these BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications.

These are likely caused by multiple parallel getrawtransaction RPC calls (probably from the lnd node) - if the txindex is enabled but hasn’t managed to process the newest block yet (as a result of a general slowness / unresponsiveness due to memory shortage), it will log this message. So I think that these are probably not the cause of the memory problem, but appear as a result of it.

either use std::move or really use .copy() if a copy is needed.

Style-wise I found it a bit unfortunate, because it forces code to either specify move or copy explicitly. (There is some beauty to rust, in that it implicitly moves)

Can you explain this? It has a move constructor, …

I think you are correct here, the compiler implicitly generates move-contructors which are enough to fix this issue, if combined with appropriate std::move. I did not think that was the case, but I was using a non-reproducible test. I made my test reproducible and tested:

  1. v27.0
  2. addition of std::move only
  3. addition of std::move + excplicit move constructor(s)
  4. addition of std::move + excplicit move constructor(s) + LD_PRELOAD=mimalloc (just for my own interest)

rest-comparison

The test script requests every block from (0 - 844,000) % 5,000, and measures the RSS after each call.

The UniValue created inside blockToJSON is ~40MB worst-case, but it is then copied into a second UniValue, doubling the required heap size.

I had a PR somewhere where I removed the copy constructor and made it mandatory to call a .copy(). Then you’d get a compile error wherever a copy happens, and then can either use std::move or really use .copy() if a copy is needed.

That’s nice because the compiler finds all problems automatically, but requires to touch quite a lot of places.

And if I understand correctly, it is then copied again here since UniValue doesn’t have a move constructor. So improving UniValue move semantics seems to make sense here indeed.

Can you explain this? It has a move constructor, at least the following compiles for me:

diff --git a/src/univalue/lib/univalue.cpp b/src/univalue/lib/univalue.cpp
index 656d2e8203..b3a33f36ac 100644
--- a/src/univalue/lib/univalue.cpp
+++ b/src/univalue/lib/univalue.cpp
@@ -125,6 +125,7 @@ void UniValue::pushKVEnd(std::string key, UniValue val)
 
 void UniValue::pushKV(std::string key, UniValue val)
 {
+static_assert(std::is_move_constructible_v<UniValue>);
     checkType(VOBJ);
 
     size_t idx;

The UniValue created inside blockToJSON is ~40MB worst-case, but it is then copied into a second UniValue, doubling the required heap size.

I presume this can be fixed by adding a std::move, but I haven’t tested it.

I think setting this to 1 will give significantly worse performance for most users though?

Seems like it would be better to add it to the documentation if anything…

So far my results with MALLOC_ARENA_MAX=1 seem to be good, I’ve been comfortably under 2GB RAM every time I’ve checked. I added it as an env var in my systemd configuration which I copied and modified from the one in this repo, I figure it ought to have that env var as well. I’ll open a PR to add it.

I’m running 24.0.1 and I tried with various rpcthreads values 50, 100, 150.

I could solve the issue by setting the below env var.

By default, since glibc 2.10, the C library will create up to two heap arenas per core. This is known to cause excessive memory usage in some scenarios. To avoid this we have to set

MALLOC_ARENA_MAX=1

without this the usage grows beyond 36GB, with this it stays under 2GB normally.

ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce- memory.md#linux-specific

FYI, I am now using v24.0.1 with rpcthreads=64 as mentioned above, and I still see this issue although it still appears to be slow and unpredictable. I just checked and bitcoind is currently using close to 4 GB RAM, which is a lot higher than it would be if I’d restarted the node within the past few hours or even days.

I increased the rpcthreads setting to make sure bitcoind plays nice with lnd, but I don’t expect that I actually need it that high. I can switch to the default rpcthreads value and see if this issue goes away, I’ll report back later.

I think I’ve seen the same thing; but mostly for me it just looks like “bitcoind has a bunch of memory in swap and it takes a while to deallocate when shutting down” so I haven’t been able to get any insight into what’s actually going on. I think the same memory leak may show up in liquid/elements, but with higher severity (due to the higher block rate)

EDIT: maybe worth noting that the “takes a while to deallocate” occurs after Shutdown() has finished