bitcoin: bitcoind 29.0 much slower than 28.0 on my system: cause found

I always compile the bitcoin software myself (Ubuntu 24.04.2 LTS). For bitcoin-29.0 I compiled it (of course) for the first time with cmake (with and without berkeley-db).

For both (with and without berkeley-db), bitcoind-29.0 is much slower than bitcoin-28.0 which gives me problems on my server.

To investigate I wrote a shell script that determines the time difference in seconds between “Saw new header” and “UpdateTip”:

2025-05-09T05:24:11Z Saw new header hash=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d height=895920
2025-05-09T05:24:11Z Saw new cmpctblock header hash=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d peer=5197
2025-05-09T05:24:12Z UpdateTip: new best=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d height=895920 version=0x2a4aa000 log2_work=95.598602 tx=1188522869 date='2025-05-09T05:23:59Z' progress=1.000000 cache=216.6MiB(1587712txo)

which in this bitcoind-28.0 example is only 1 second. This is typical for bitcoind-28.0 on my system where it is between 0 and 1 seconds with only very occasionally a bit longer. Now for bitcoind-29.0 it’s much longer:

Block nr: delta t in seconds 895579: 3 895580: 3 895581: 4 895582: 2 895583: 16 895584: 65 895585: 8 895586: 280 895587: 81 895588: 133 895589: 124 895590: 3 895591: 73 895592: 153 895593: 284 895594: 528 895595: 17 895596: 8 895597: 4 895598: 2 895599: 3 895600: 4 895601: 6 895602: 3 895603: 2 895604: 5 895605: 5 895606: 34 895607: 2 895608: 6

This is true whether it’s compiled with or without berkeley-db. So mometimes it takes several minutes!

So I did some investigations. The first thing I found was that my bitcoind-29.0 and others are much bigger than the pre-compiled ones:

Precompiled 29.0 (du in MB): 3 bitcoin-cli 16 bitcoind 42 bitcoin-qt 5 bitcoin-tx 3 bitcoin-util 10 bitcoin-wallet 28 test_bitcoin

For my self-compiled 29.0: 21 bitcoin-cli 273 bitcoind 47 bitcoin-tx 21 bitcoin-util 134 bitcoin-wallet 511 test_bitcoin

Although normally I use some compiler options, here for comparison compiled according to the docs: (cmake -B build; cmake --build build; cmake --install build)

Both (my and pre-compiled) test_bitcoin work fine. Note that also my (not too slow) bitcoind-28.0 (etc) is about the same too big a size (269 MB) as my bitcoind-29.0. So I had this before but never noticed.

So my first question before I investigate any further is: how come my bitcoind is so big compared to the pre-compiled one?

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 69 (27 by maintainers)

Most upvoted comments

it would be really interesting to see how often you get an UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli ... CLN warning if you just let version 29.0 run without a reindex. The expectation is that should decrease over time. I now run without setting bitcoin-rpcclienttimeout in the CLN config file and (after a reindex) get about 1 warning a day and the delay is usually a little over 10 seconds (between 10-12 seconds).

@hMsats: I have never reïndexed.

Do you know if you ever got this CLN warning before switching to 29.0?

  • I got the UNUSUAL message complaining about bitcoin-cli taking long than 10 seconds…
    • …throughout 2020: 5093 times.
    • …throughout 2021: 2071 times.
    • …throughout 2022: 788 times.
    • …throughout 2023: 4129 times.
    • …throughout 2024: 10916 times.
    • …so far in 2025: 8656 times — so it appears to be on its worst pace ever.
  • I got the BROKEN message due to bitcoin-cli taking longer than 60 seconds…
    • …throughout 2020: 0 times.
    • …throughout 2021: 1 time.
    • …throughout 2022: 0 times.
    • …throughout 2023: 0 times.
    • …throughout 2024: 3 times.
    • …so far in 2025: 5 times.
  • I upgraded to Bitcoin Core 29.0 on 12 April 2025. After the upgrade…
    • I got the UNUSUAL message 1929 times.
    • I got the BROKEN message 3 times, but I upped my timeout to 300 seconds on 6 June (and didn’t get any more BROKEN messages after that).
    • The most recent UNUSUAL message reporting a bitcoin-cli latency greater than 60 seconds (which would have resulted in CLN death if not for my increased timeout setting) was only two days ago:
      2025-06-21T00:49:35.250Z UNUSUAL plugin-bcli: bitcoin-cli: finished bitcoin-cli -datadir=/var/lib/bitcoind -rpcclienttimeout=300 getblockhash 902113 (90895 ms)
      

What’s your system setup

An ancient Intel Core 2 Quad Q6600 (Kentsfield) with 8 GB RAM. Bitcoin Core’s chainstate LevelDB database resides on a Linux mdraid striped (“RAID0”) pair of SATA SSDs (a Crucial MX500 and a Samsung 860 EVO) attached to the motherboard’s Intel NM10/ICH7 SATA controller.

and are you running with txindex=1?

No.

$ grep '^\w' /etc/bitcoin/bitcoin.conf
blocknotify=/var/lib/bitcoind/blocknotify.sh %s
dbcache=32
maxmempool=64
mempoolexpiry=72
listenonion=0
walletnotify=/var/lib/bitcoind/walletnotify.sh %s %w %b %h
walletrbf=1
whitelistforcerelay=1
rpcthreads=16

I have measured the cost of switching from 2MiB max_file_size to 32MiB from 700k blocks to 888,888:

Image

Details
COMMIT_2MIB="c9417a59ee7b65d9dd3352c55f0e414d5dbdb7af"; COMMIT_32MIB="370c59261269fd9043674e0f4fd782a89e724473"; \            
STOP1=700000; STOP2=888888; DBCACHE=2500; \                                                                                                                                               
BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; mkdir -p $LOG_DIR; \                                                                              
BUILD_AND_RUN() { \                                                                                                                                                                       
  local COMMIT=$1 STOP=$2 EXTRA=$3; \                                                                                                                                                     
  git checkout $COMMIT && git clean -fxd && git reset --hard && \                                                                                                                         
  cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C build bitcoind && \                                                                                                      
  ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP -dbcache=$DBCACHE -blocksonly -printtoconsole=0 -debug=bench -debug=leveldb $EXTRA; \                                       
} && \                                                                                                                                                                                    
BUILD_AND_RUN $COMMIT_2MIB $STOP2 "" && \
echo -e "\n-- Test 1: 2MiB to 2MiB --" && \
BUILD_AND_RUN $COMMIT_2MIB $STOP1 "-connect=0 -reindex-chainstate" && BUILD_AND_RUN $COMMIT_2MIB $STOP2 "-connect=0" && \
cp $DATA_DIR/debug.log $LOG_DIR/debug-2-2-${COMMIT_2MIB:0:8}-$(date +%s).log && \
echo -e "\n-- Test 2: 2MiB to 32MiB (conversion) --" && \
BUILD_AND_RUN $COMMIT_2MIB $STOP1 "-connect=0 -reindex-chainstate" && BUILD_AND_RUN $COMMIT_32MIB $STOP2 "-connect=0" && \
cp $DATA_DIR/debug.log $LOG_DIR/debug-2-32-${COMMIT_32MIB:0:8}-$(date +%s).log && \
echo -e "\nAll tests complete!" && ls -la $LOG_DIR/debug-2-*.log
HEAD is now at c9417a59ee DBWRAPPER_MAX_FILE_SIZE = 2 << 20

The resulting logs confirm that LevelDB is indeed compacting to bigger files after restarting with 32MiB starting from 700k blocks:

2MiB -> 2MiB

Image

Image

2MiB -> 32MiB

Image

Image

This indicates that full compaction will likely take some time. It also indicates that even this is faster than just the 2MiB file, even though there are more waits in the 2 -> 32 MiB scenario:

$ cat debug-2-2-c9417a59-1749262598.log | grep 'Too many L0 files' | wc -l
   28327
$ cat debug-2-32-370c5926-1749282484.log | grep 'Too many L0 files' | wc -l
   34930

you can also see in the plots that compacting many 2MiB to 32MiB can sometimes indeed temporarily have bigger spikes:

2MiB -> 2MiB

Image

2MiB -> 32MiB

Image


So @andrewtoth’s intuition seems to have been correct, I’m still waiting for your assessment, but otherwise I consider this mystery solved - I will try to find a way to add a warning if we detect the user still has 2 MiB blocks to avoid these surprises in the future.

  1. & 2)

MAX_BLOCK_DB_CACHE sets block_cache which is used to configure leveldb::Options.block_cache and .write_buffer_size. It’s not strictly related to max_file_size - see https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/db/db_impl.cc#L104-L105.

2 MiB is a clear winner

@andrewtoth just had a brilliant realization that I hadn’t thought of: you’ve just switched to v29, so you will still need to convert your 2 MiB files to 32 MiB files which requires heavy compaction at first: basically your whole index has to be rewritten. It’s a temporary growing pain (literally). I’ll measure this properly and if the effect is as obvious as we think it is, we may want to add a warning to 29.1 if we detect small leveldb file sizes. So if you want an even clearer winner, you could probably leave it for the night with v29 to let it compact - and tell us if that fixed the issue.

shouldn’t options.block_cache and options.write_buffer_size be increased?

I have experimented a lot with these and they all just slowed down IBD for me. Definitely let us know if it’s different on your system.

Compaction could theoretically be a problem, but my understanding is that it would just mean that a few spikes are larger, most other processing should be significantly faster. Is this your only complain? On average this made IBD and block processing in general ~30% faster - but there’s a tradeoff, it also means that some blocks will be significantly slower now (when compaction cannot be delayed anymore). So we would expect more variation between blocks in 29, but most of the processing should be faster now on average - it’s why I have recommended bumping it from 2 to 8 or 16 in https://github.com/bitcoin/bitcoin/pull/30039#issuecomment-2397229720, since that would likely have caused smaller spikes, while retaining most of the speedup on average. Luckily we can still do that in a minor update theoretically, LevelDB allows changing this size dynamically (after compaction the new file size limit is enforced, regardless of the input file sizes).

Thanks for double checking these, I will run a few measurements in the next weeks and plot the average new-header-to-UpdateTip times to understand this better.

Yes, since the requests are io bound they will mostly be waiting concurrently and not using much CPU.

@whitslack thanks a lot for the elaborate answer! I get the impression that although CLN was already having some problems, it deteriorated after upgrading bitcoind to 29.0 because you got the CLN BROKEN message more often, even after running 29.0 for many weeks

I too had to increase CLN’s bitcoin-rpcclienttimeout and bitcoin-retry-timeout to 300 seconds after upgrading to Bitcoin Core 29.0 because CLN kept dying due to Bitcoin RPCs taking multiple minutes to complete.

A general question: Why does Bitcoin Core stop responding to RPCs while it is validating a new chain tip? Shouldn’t block validation be concurrent with handling RPCs? (The RPCs would simply query the chainstate as of the previous tip until the new tip finishes being validated.)

@hMsats are you aware of the -blocksdir argument for bitcoind? It stores the blocks at <blocksdir>/.bitcoin/blocks and then the rest of the data is at <datadir>/.bitcoin. So you don’t have to use symlinks, you can just do bitcoind -blocksdir=<external SSD>/.bitcoin -datadir=<internal SSD>/.bitcoin or omit -datadir altogether since that would be the default location.

External USB disk controllers have far lower IO operations/s second than internal ones. Even if the sequential read/write speed seems great, they’re really not appropriate for heavy database loads.

Thanks for following up. Not sure I fully understand the implications of all the changes (hardware and software), but my understanding is that the problems is solved. I’m not sure the warning is needed, but we can reconsider that if more people complain.

@l0rinc in a few days a 4 tb ssd will arrive. I will copy the data and do a -reindex (with 32 MiB) using a different machine. That way my server will be up most of the time. It’s a a good way to test the ssd and the blocks directory.

Ok, just wanted to point out you don’t need a -reindex just for recreating the txindex. -reindex will wipe all your block index and chainstate as well.

@andrewtoth I just realized I shouldn’t remove indexes because a user that switches from a lower version to 29.0 wouldn’t do that and I want to see what happens in the most common use case.

Does increasing the file sizes in indexes/txindex require a -reindex?

@hMsats -reindex is overkill for recreating the txindex. Shut down the node, delete .bitcoin/indexes and start up, and only the txindex will be recreated.

Another observation: the files in blocks/index and chainstate are about 32 MiB now but the files in indexes/txindex are still at 2 MiB. Does increasing the file sizes in indexes/txindex require a -reindex?

Some of the files have now moved to 32MiB also in .bitcoin/indexes/txindex: diff_indexes_txindex_du.txt.

Thanks again for all this information, although it’s not always easy to understand what the precise consequences are for my particular system.

I had to learn the hard way that the command dd is very dangerous, which caused quite a bit of a delay but I’m up and running again. My bitcoind, Fulcrum and 2 CLN nodes are now running for 6 hours with max_file_size=32MiB for bitcoind (starting from max_file_size=2MiB ) . Here are some preliminary results for the different file sizes for 2MiB and 32MiB (unix diff command):

diff_blocks_index_du.txt diff_blocks_index_ls-l.txt diff_chainstate_du.txt diff_chainstate_ls-l.txt diff_indexes_txindex_du.txt diff_indexes_txindex_ls-l.txt

In (approximately) the first few hours only (approximately) half the chainstate files were converted to 32 MiB.

I have a method that informs me when (if ever) the complaints from Fulcrum and CLN are over. Tomorrow I will have a look if that happened.

Whatever the outcome, a warning will be very useful for a user because many users will assume that the changes as described in the 29.0 release won’t affect them when they upgrade but it might.

I was too forgiving in my search for the problematic commit and did the search again holding on to my original low cpu system settings. A (very) long story short, the problematic commit for my system is 097c66f, where the LevelDB max file size is increase to 32 MiB.

I tested that my system ran perfectly again on (almost) master (v29.99.0-14c16e81598a) removing the code changes by this commit (so removing: options.max_file_size = std::max(options.max_file_size, DBWRAPPER_MAX_FILE_SIZE); and static const size_t DBWRAPPER_MAX_FILE_SIZE = 32 << 20; // 32 MiB from dbwrapper.cpp and dbwrapper.h)

So to keep things as short as possible:

  • running bitcoind (low cpu) without the LevelDB increase, block verification times are 0 or 1 seconds (except for the very beginning) and both Fulcrum (electrum server) and Core Lightning (CLN) never complain
  • running bitcoind (low cpu) with the LevelDB increase, verification times are sometimes also 0 and 1 seconds but occasionally increase to much longer times (compactification?) and Fulcrum (see: Lost connection to bitcoind) and CLN (see: UNUSUAL plugin-bcli) complain a lot.
  • this situation didn’t improve (much) by not reducing my cpu speed. See bitcoin, times, Fulcrum , CLN

So there seems to be more than meets the eye in this reasonable LevelDB file size increase, maybe due to compactification times but maybe there’s more going on.

I understand that this setting won’t be changed based on the complains of one user but maybe others will see similar issues. Should the LevelDB max file size become a optional parameter in bitcoin.conf?

So I’m going to run bitcoind 29.0 without the LevelDB file size increase on my server.

bitcoin.conf: txindex=1 rpcthreads=4 rpcworkqueue=64 dbcache=2500 maxconnections=100

System info:

Acer Aspire E1-572, 64 bits 8 GB RAM, 4x Intel 1.6 GHz (2.6 GHz max) CPU capped at 1.3 GHz to reduce fan usage

2 TB external SSD

Ubuntu 24.04.2 LTS swapoff -a vm.swappiness=0

Settings for low cpu:

sudo cpufreq-set -c 0 -u 1300000 sudo cpufreq-set -c 1 -u 1300000 sudo cpufreq-set -c 2 -u 1300000 sudo cpufreq-set -c 3 -u 1300000

sudo cpufreq-set -c 0 -g powersave sudo cpufreq-set -c 1 -g powersave sudo cpufreq-set -c 2 -g powersave sudo cpufreq-set -c 3 -g powersave

Settings for high/normal cpu: sudo cpufreq-set -c 0 -u 2600000 sudo cpufreq-set -c 1 -u 2600000 sudo cpufreq-set -c 2 -u 2600000 sudo cpufreq-set -c 3 -u 2600000

sudo cpufreq-set -c 0 -g performance sudo cpufreq-set -c 1 -g performance sudo cpufreq-set -c 2 -g performance sudo cpufreq-set -c 3 -g performance

Couldn’t find a commit error (luckily). Improved a few things on my server and now it seems to be working well:

Removed permitbaremultisig=0, datacarrier=0, increased my reduced cpu speed, give bitcoind more time before starting public electrum server Fulcrum and Core Lightning (CLN). Maybe 29.0 it a bit more demanding than 28.0. Sorry for the confusion and thanks for all the help!

For now I think something bad happens between commit:

52fd1511a774d1ff9e747b2ce88aa1fcb778ced8

and

0a159f0914775756dcac2d9fa7fe4e4d4e70ba0c

I’ll try to narrow it down.

In validation.cpp else if (!check()) was changed into else if (auto result = check(); result.has_value()). Shouldn’t that be else if (auto result = check(); !result.has_value())?

Why do you think that’s the error?

If this were incorrect, you wouldn’t be seeing that validation be slow; it would instead be failing on a bunch of blocks.

The code on that line is correct. It was changed from a bool in 28.x to a std::optional<std::pair<ScriptError, std::string>> in 29.x. If the return value has a value, then that means an error occurred and the value contained is the error. On success, it returns no value.

@maflcko I found time to run 29.0 with the following debug catagories:

debug=bench
debug=blockstorage
debug=cmpctblock
debug=tor
debug=validation
debug=rpc
debug=zmq

You can find it here and more system information on my website.

There are background callers. Besides a full node I run: Fulcrum (public electrum server), 2 CLN lightning nodes, electrum personal server. It’s maybe a lot but 28.0 has no problems with it. At the beginning of my debug test I started these as well.

@maflcko I will but it will take some time as I’m on vacation right now but will investigate when I have time.

Created bitcoind via:

cmake -B build -DWITH_ZMQ=ON -DBUILD_GUI=ON

Settings (same as for 28.0):

txindex=1
rpcthreads=4
rpcworkqueue=64
dbcache=2500
maxconnections=100
permitbaremultisig=0
datacarrier=0