bitcoin: Slow memory leak in v22.0?

I’ve been noticing gradually increasing memory usage bitcoin core v22.0. I’ve been running this node for a couple of years with the same hardware and general setup and have only noticed this in the past few months, around the time I upgraded to v22, which makes me suspect it may be related.

I’m seeing bitcoind memory usage slowly climb over weeks/months of uptime. Right now it’s at 3.4 GB memory usage and as I recall I’ve seen it over 5 GB before in an incident where it caused the machine to run out of RAM, which is when I first noticed this. If I restart bitcoind, RAM usage goes back to a normal level and stays there for a while.

I do recall seeing a lot of BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications messages in the logs when RAM usage was over 5GB and my node was barely responsive to RPC calls, but I’m not sure if this has anything to do with the issue.

I’m using dbcache=1024 which would explain some of the RAM usage but not all of it.

Other config settings that may be relevant:

txindex=1
peerblockfilters=1
blockfilterindex=1

maxconnections=100

rpcthreads=64
rpcworkqueue=256
rpcservertimeout=120

zmqpubrawblockhwm=100000
zmqpubrawtxhwm=100000

Otherwise, I’m using this node to run an lnd node (with zmq block/tx notifications) and electrumx server. There’s a medium-sized watch-only wallet that’s loaded. There’s also a small script I run with blocknotify.

This is all running on an RPI4 with 8GB RAM, using raspbian buster w/ the OS and block indexes on an SSD.

I’d be interested if others have seen anything similar out of their nodes or if perhaps this RAM usage is normal given how I’m using it, but I am pretty sure this wasn’t happening a year ago with similar usage patterns.

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 16 (14 by maintainers)

Commits related to this issue

init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to... — committed to sangaman/bitcoin by sangaman a year ago
init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to t... — committed to sangaman/bitcoin by sangaman a year ago
init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to t... — committed to sangaman/bitcoin by sangaman a year ago
init: add MALLOC_ARENA_MAX=1 to systemd This adds the `MALLOC_ARENA_MAX=1` environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to t... — committed to sangaman/bitcoin by sangaman a year ago

Most upvoted comments

Looking at the logs, there are tons of these BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications.

These are likely caused by multiple parallel getrawtransaction RPC calls (probably from the lnd node) - if the txindex is enabled but hasn’t managed to process the newest block yet (as a result of a general slowness / unresponsiveness due to memory shortage), it will log this message. So I think that these are probably not the cause of the memory problem, but appear as a result of it.

mzumsande on Mar 15, 2022

I think setting this to 1 will give significantly worse performance for most users though?

Seems like it would be better to add it to the documentation if anything…

willcl-ark on May 12, 2023

So far my results with MALLOC_ARENA_MAX=1 seem to be good, I’ve been comfortably under 2GB RAM every time I’ve checked. I added it as an env var in my systemd configuration which I copied and modified from the one in this repo, I figure it ought to have that env var as well. I’ll open a PR to add it.

sangaman on May 12, 2023

FYI, I am now using v24.0.1 with rpcthreads=64 as mentioned above, and I still see this issue although it still appears to be slow and unpredictable. I just checked and bitcoind is currently using close to 4 GB RAM, which is a lot higher than it would be if I’d restarted the node within the past few hours or even days.

I increased the rpcthreads setting to make sure bitcoind plays nice with lnd, but I don’t expect that I actually need it that high. I can switch to the default rpcthreads value and see if this issue goes away, I’ll report back later.

sangaman on May 4, 2023

I think I’ve seen the same thing; but mostly for me it just looks like “bitcoind has a bunch of memory in swap and it takes a while to deallocate when shutting down” so I haven’t been able to get any insight into what’s actually going on. I think the same memory leak may show up in liquid/elements, but with higher severity (due to the higher block rate)

EDIT: maybe worth noting that the “takes a while to deallocate” occurs after Shutdown() has finished

ajtowns on Mar 15, 2022