bitcoin: Slow memory leak in v22.0?
I’ve been noticing gradually increasing memory usage bitcoin core v22.0. I’ve been running this node for a couple of years with the same hardware and general setup and have only noticed this in the past few months, around the time I upgraded to v22, which makes me suspect it may be related.
I’m seeing bitcoind memory usage slowly climb over weeks/months of uptime. Right now it’s at 3.4 GB memory usage and as I recall I’ve seen it over 5 GB before in an incident where it caused the machine to run out of RAM, which is when I first noticed this. If I restart bitcoind, RAM usage goes back to a normal level and stays there for a while.
I do recall seeing a lot of BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications messages in the logs when RAM usage was over 5GB and my node was barely responsive to RPC calls, but I’m not sure if this has anything to do with the issue.
I’m using dbcache=1024 which would explain some of the RAM usage but not all of it.
Other config settings that may be relevant:
txindex=1
peerblockfilters=1
blockfilterindex=1
maxconnections=100
rpcthreads=64
rpcworkqueue=256
rpcservertimeout=120
zmqpubrawblockhwm=100000
zmqpubrawtxhwm=100000
Otherwise, I’m using this node to run an lnd node (with zmq block/tx notifications) and electrumx server. There’s a medium-sized watch-only wallet that’s loaded. There’s also a small script I run with blocknotify.
This is all running on an RPI4 with 8GB RAM, using raspbian buster w/ the OS and block indexes on an SSD.
I’d be interested if others have seen anything similar out of their nodes or if perhaps this RAM usage is normal given how I’m using it, but I am pretty sure this wasn’t happening a year ago with similar usage patterns.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 25 (23 by maintainers)
Commits related to this issue
- init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to... — committed to sangaman/bitcoin by sangaman 4 years ago
- init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to t... — committed to sangaman/bitcoin by sangaman 4 years ago
- init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to t... — committed to sangaman/bitcoin by sangaman 4 years ago
- init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to t... — committed to sangaman/bitcoin by sangaman 4 years ago
- Merge bitcoin/bitcoin#30094: rpc: move UniValue in blockToJSON b77bad309e92f176f340598eec056eb7bff86f5f rpc: move UniValue in blockToJSON (willcl-ark) Pull request description: Fixes: #24542 Fi... — committed to theuni/bitcoin by ryanofsky 3 years ago
OK, I think I have pretty much identified the root cause here.
Calling
blockToJSONon a maximum-sized block allocates 70-80MB on the heap during runtime.There is then an additional 15-20MB allocated for
strJSON.After these allocations are made they then seem to be re-used when possible in the future, so long as subsequent blocks are small enough to fit. This totals up to the ~100MB RSS increase observed sometimes when getting blocks via REST.
In the event that blocks are fetched in series in increasing sizes, repeated allocations (of increasing sizes) may be made, causing even higher apparent resource usage.
The cause seems to be two-fold:
UniValue’s are not a particularly compact representation (can’t do that much about this, except perhaps we could write to string directly for this method.)UniValuecreated insideblockToJSONis ~40MB worst-case, but it is then copied into a secondUniValue, doubling the required heap size.It appears that this has been investgated previously (see also here) in search of better speed, but having a move semantic for UniValue would also shave off half of this function’s allocation requirement.
Thanks @stickies-v for helping me investigate
These are likely caused by multiple parallel
getrawtransactionRPC calls (probably from the lnd node) - if the txindex is enabled but hasn’t managed to process the newest block yet (as a result of a general slowness / unresponsiveness due to memory shortage), it will log this message. So I think that these are probably not the cause of the memory problem, but appear as a result of it.Style-wise I found it a bit unfortunate, because it forces code to either specify
moveorcopyexplicitly. (There is some beauty to rust, in that it implicitly moves)I think you are correct here, the compiler implicitly generates move-contructors which are enough to fix this issue, if combined with appropriate
std::move. I did not think that was the case, but I was using a non-reproducible test. I made my test reproducible and tested:std::moveonlystd::move+ excplicit move constructor(s)std::move+ excplicit move constructor(s) + LD_PRELOAD=mimalloc (just for my own interest)The test script requests every block from (0 - 844,000) % 5,000, and measures the RSS after each call.
I had a PR somewhere where I removed the copy constructor and made it mandatory to call a
.copy(). Then you’d get a compile error wherever a copy happens, and then can either usestd::moveor really use.copy()if a copy is needed.That’s nice because the compiler finds all problems automatically, but requires to touch quite a lot of places.
Can you explain this? It has a move constructor, at least the following compiles for me:
I presume this can be fixed by adding a
std::move, but I haven’t tested it.I think setting this to 1 will give significantly worse performance for most users though?
Seems like it would be better to add it to the documentation if anything…
So far my results with
MALLOC_ARENA_MAX=1seem to be good, I’ve been comfortably under 2GB RAM every time I’ve checked. I added it as an env var in my systemd configuration which I copied and modified from the one in this repo, I figure it ought to have that env var as well. I’ll open a PR to add it.I’m running
24.0.1and I tried with variousrpcthreadsvalues50, 100, 150.I could solve the issue by setting the below env var.
By default, since glibc 2.10, the C library will create up to two heap arenas per core. This is known to cause excessive memory usage in some scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1without this the usage grows beyond 36GB, with this it stays under 2GB normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce- memory.md#linux-specific
FYI, I am now using v24.0.1 with
rpcthreads=64as mentioned above, and I still see this issue although it still appears to be slow and unpredictable. I just checked and bitcoind is currently using close to 4 GB RAM, which is a lot higher than it would be if I’d restarted the node within the past few hours or even days.I increased the
rpcthreadssetting to make sure bitcoind plays nice with lnd, but I don’t expect that I actually need it that high. I can switch to the defaultrpcthreadsvalue and see if this issue goes away, I’ll report back later.I think I’ve seen the same thing; but mostly for me it just looks like “bitcoind has a bunch of memory in swap and it takes a while to deallocate when shutting down” so I haven’t been able to get any insight into what’s actually going on. I think the same memory leak may show up in liquid/elements, but with higher severity (due to the higher block rate)
EDIT: maybe worth noting that the “takes a while to deallocate” occurs after
Shutdown()has finished