bitcoin: bitcoind 29.0 much slower than 28.0 on my system: cause found
I always compile the bitcoin software myself (Ubuntu 24.04.2 LTS). For bitcoin-29.0 I compiled it (of course) for the first time with cmake (with and without berkeley-db).
For both (with and without berkeley-db), bitcoind-29.0 is much slower than bitcoin-28.0 which gives me problems on my server.
To investigate I wrote a shell script that determines the time difference in seconds between “Saw new header” and “UpdateTip”:
2025-05-09T05:24:11Z Saw new header hash=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d height=895920
2025-05-09T05:24:11Z Saw new cmpctblock header hash=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d peer=5197
2025-05-09T05:24:12Z UpdateTip: new best=0000000000000000000213e630619be8945d471d06b0395fb6adca797877527d height=895920 version=0x2a4aa000 log2_work=95.598602 tx=1188522869 date='2025-05-09T05:23:59Z' progress=1.000000 cache=216.6MiB(1587712txo)
which in this bitcoind-28.0 example is only 1 second. This is typical for bitcoind-28.0 on my system where it is between 0 and 1 seconds with only very occasionally a bit longer. Now for bitcoind-29.0 it’s much longer:
Block nr: delta t in seconds 895579: 3 895580: 3 895581: 4 895582: 2 895583: 16 895584: 65 895585: 8 895586: 280 895587: 81 895588: 133 895589: 124 895590: 3 895591: 73 895592: 153 895593: 284 895594: 528 895595: 17 895596: 8 895597: 4 895598: 2 895599: 3 895600: 4 895601: 6 895602: 3 895603: 2 895604: 5 895605: 5 895606: 34 895607: 2 895608: 6
This is true whether it’s compiled with or without berkeley-db. So mometimes it takes several minutes!
So I did some investigations. The first thing I found was that my bitcoind-29.0 and others are much bigger than the pre-compiled ones:
Precompiled 29.0 (du in MB):
3 bitcoin-cli
16 bitcoind
42 bitcoin-qt
5 bitcoin-tx
3 bitcoin-util
10 bitcoin-wallet
28 test_bitcoin
For my self-compiled 29.0:
21 bitcoin-cli
273 bitcoind
47 bitcoin-tx
21 bitcoin-util
134 bitcoin-wallet
511 test_bitcoin
Although normally I use some compiler options, here for comparison compiled according to the docs: (cmake -B build; cmake --build build; cmake --install build)
Both (my and pre-compiled) test_bitcoin work fine. Note that also my (not too slow) bitcoind-28.0 (etc) is about the same too big a size (269 MB) as my bitcoind-29.0. So I had this before but never noticed.
So my first question before I investigate any further is: how come my bitcoind is so big compared to the pre-compiled one?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 69 (27 by maintainers)
@hMsats: I have never reïndexed.
UNUSUALmessage complaining aboutbitcoin-clitaking long than 10 seconds…BROKENmessage due tobitcoin-clitaking longer than 60 seconds…UNUSUALmessage 1929 times.BROKENmessage 3 times, but I upped my timeout to 300 seconds on 6 June (and didn’t get any moreBROKENmessages after that).UNUSUALmessage reporting abitcoin-clilatency greater than 60 seconds (which would have resulted in CLN death if not for my increased timeout setting) was only two days ago:An ancient Intel Core 2 Quad Q6600 (Kentsfield) with 8 GB RAM. Bitcoin Core’s
chainstateLevelDB database resides on a Linux mdraid striped (“RAID0”) pair of SATA SSDs (a Crucial MX500 and a Samsung 860 EVO) attached to the motherboard’s Intel NM10/ICH7 SATA controller.No.
I have measured the cost of switching from
2MiBmax_file_sizeto32MiBfrom 700k blocks to 888,888:Details
The resulting logs confirm that LevelDB is indeed compacting to bigger files after restarting with 32MiB starting from 700k blocks:
This indicates that full compaction will likely take some time. It also indicates that even this is faster than just the 2MiB file, even though there are more waits in the 2 -> 32 MiB scenario:
you can also see in the plots that compacting many 2MiB to 32MiB can sometimes indeed temporarily have bigger spikes:
So @andrewtoth’s intuition seems to have been correct, I’m still waiting for your assessment, but otherwise I consider this mystery solved - I will try to find a way to add a warning if we detect the user still has 2 MiB blocks to avoid these surprises in the future.
MAX_BLOCK_DB_CACHEsetsblock_cachewhich is used to configureleveldb::Options.block_cacheand.write_buffer_size. It’s not strictly related tomax_file_size- see https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/db/db_impl.cc#L104-L105.@andrewtoth just had a brilliant realization that I hadn’t thought of: you’ve just switched to v29, so you will still need to convert your 2 MiB files to 32 MiB files which requires heavy compaction at first: basically your whole index has to be rewritten. It’s a temporary growing pain (literally). I’ll measure this properly and if the effect is as obvious as we think it is, we may want to add a warning to 29.1 if we detect small leveldb file sizes. So if you want an even clearer winner, you could probably leave it for the night with v29 to let it compact - and tell us if that fixed the issue.
I have experimented a lot with these and they all just slowed down IBD for me. Definitely let us know if it’s different on your system.
Compaction could theoretically be a problem, but my understanding is that it would just mean that a few spikes are larger, most other processing should be significantly faster. Is this your only complain? On average this made IBD and block processing in general ~30% faster - but there’s a tradeoff, it also means that some blocks will be significantly slower now (when compaction cannot be delayed anymore). So we would expect more variation between blocks in 29, but most of the processing should be faster now on average - it’s why I have recommended bumping it from 2 to 8 or 16 in https://github.com/bitcoin/bitcoin/pull/30039#issuecomment-2397229720, since that would likely have caused smaller spikes, while retaining most of the speedup on average. Luckily we can still do that in a minor update theoretically, LevelDB allows changing this size dynamically (after compaction the new file size limit is enforced, regardless of the input file sizes).
Thanks for double checking these, I will run a few measurements in the next weeks and plot the average new-header-to-UpdateTip times to understand this better.
Yes, since the requests are io bound they will mostly be waiting concurrently and not using much CPU.
@whitslack thanks a lot for the elaborate answer! I get the impression that although CLN was already having some problems, it deteriorated after upgrading bitcoind to 29.0 because you got the CLN BROKEN message more often, even after running 29.0 for many weeks
I too had to increase CLN’s
bitcoin-rpcclienttimeoutandbitcoin-retry-timeoutto 300 seconds after upgrading to Bitcoin Core 29.0 because CLN kept dying due to Bitcoin RPCs taking multiple minutes to complete.A general question: Why does Bitcoin Core stop responding to RPCs while it is validating a new chain tip? Shouldn’t block validation be concurrent with handling RPCs? (The RPCs would simply query the chainstate as of the previous tip until the new tip finishes being validated.)
@hMsats are you aware of the
-blocksdirargument for bitcoind? It stores the blocks at<blocksdir>/.bitcoin/blocksand then the rest of the data is at<datadir>/.bitcoin. So you don’t have to use symlinks, you can just dobitcoind -blocksdir=<external SSD>/.bitcoin -datadir=<internal SSD>/.bitcoinor omit-datadiraltogether since that would be the default location.External USB disk controllers have far lower IO operations/s second than internal ones. Even if the sequential read/write speed seems great, they’re really not appropriate for heavy database loads.
Thanks for following up. Not sure I fully understand the implications of all the changes (hardware and software), but my understanding is that the problems is solved. I’m not sure the warning is needed, but we can reconsider that if more people complain.
@l0rinc in a few days a 4 tb ssd will arrive. I will copy the data and do a
-reindex(with 32 MiB) using a different machine. That way my server will be up most of the time. It’s a a good way to test the ssd and the blocks directory.Ok, just wanted to point out you don’t need a -reindex just for recreating the txindex. -reindex will wipe all your block index and chainstate as well.
@andrewtoth I just realized I shouldn’t remove
indexesbecause a user that switches from a lower version to 29.0 wouldn’t do that and I want to see what happens in the most common use case.@hMsats -reindex is overkill for recreating the txindex. Shut down the node, delete
.bitcoin/indexesand start up, and only the txindex will be recreated.Another observation: the files in blocks/index and chainstate are about 32 MiB now but the files in indexes/txindex are still at 2 MiB. Does increasing the file sizes in indexes/txindex require a -reindex?Some of the files have now moved to 32MiB also in
.bitcoin/indexes/txindex: diff_indexes_txindex_du.txt.Thanks again for all this information, although it’s not always easy to understand what the precise consequences are for my particular system.
I had to learn the hard way that the command
ddis very dangerous, which caused quite a bit of a delay but I’m up and running again. My bitcoind, Fulcrum and 2 CLN nodes are now running for 6 hours withmax_file_size=32MiBfor bitcoind (starting frommax_file_size=2MiB) . Here are some preliminary results for the different file sizes for 2MiB and 32MiB (unixdiffcommand):diff_blocks_index_du.txt diff_blocks_index_ls-l.txt diff_chainstate_du.txt diff_chainstate_ls-l.txt diff_indexes_txindex_du.txt diff_indexes_txindex_ls-l.txt
In (approximately) the first few hours only (approximately) half the chainstate files were converted to 32 MiB.
I have a method that informs me when (if ever) the complaints from Fulcrum and CLN are over. Tomorrow I will have a look if that happened.
Whatever the outcome, a warning will be very useful for a user because many users will assume that the changes as described in the 29.0 release won’t affect them when they upgrade but it might.
I was too forgiving in my search for the problematic commit and did the search again holding on to my original low cpu system settings. A (very) long story short, the problematic commit for my system is 097c66f, where the LevelDB max file size is increase to 32 MiB.
I tested that my system ran perfectly again on (almost) master (v29.99.0-14c16e81598a) removing the code changes by this commit (so removing:
options.max_file_size = std::max(options.max_file_size, DBWRAPPER_MAX_FILE_SIZE);andstatic const size_t DBWRAPPER_MAX_FILE_SIZE = 32 << 20; // 32 MiBfromdbwrapper.cppanddbwrapper.h)So to keep things as short as possible:
So there seems to be more than meets the eye in this reasonable LevelDB file size increase, maybe due to compactification times but maybe there’s more going on.
I understand that this setting won’t be changed based on the complains of one user but maybe others will see similar issues. Should the LevelDB max file size become a optional parameter in
bitcoin.conf?So I’m going to run bitcoind 29.0 without the LevelDB file size increase on my server.
bitcoin.conf: txindex=1 rpcthreads=4 rpcworkqueue=64 dbcache=2500 maxconnections=100
System info:
Acer Aspire E1-572, 64 bits 8 GB RAM, 4x Intel 1.6 GHz (2.6 GHz max) CPU capped at 1.3 GHz to reduce fan usage
2 TB external SSD
Ubuntu 24.04.2 LTS swapoff -a vm.swappiness=0
Settings for low cpu:
sudo cpufreq-set -c 0 -u 1300000 sudo cpufreq-set -c 1 -u 1300000 sudo cpufreq-set -c 2 -u 1300000 sudo cpufreq-set -c 3 -u 1300000
sudo cpufreq-set -c 0 -g powersave sudo cpufreq-set -c 1 -g powersave sudo cpufreq-set -c 2 -g powersave sudo cpufreq-set -c 3 -g powersave
Settings for high/normal cpu: sudo cpufreq-set -c 0 -u 2600000 sudo cpufreq-set -c 1 -u 2600000 sudo cpufreq-set -c 2 -u 2600000 sudo cpufreq-set -c 3 -u 2600000
sudo cpufreq-set -c 0 -g performance sudo cpufreq-set -c 1 -g performance sudo cpufreq-set -c 2 -g performance sudo cpufreq-set -c 3 -g performance
Couldn’t find a commit error (luckily). Improved a few things on my server and now it seems to be working well:
Removed permitbaremultisig=0, datacarrier=0, increased my reduced cpu speed, give bitcoind more time before starting public electrum server Fulcrum and Core Lightning (CLN). Maybe 29.0 it a bit more demanding than 28.0. Sorry for the confusion and thanks for all the help!
For now I think something bad happens between commit:
52fd1511a774d1ff9e747b2ce88aa1fcb778ced8
and
0a159f0914775756dcac2d9fa7fe4e4d4e70ba0c
I’ll try to narrow it down.
Why do you think that’s the error?
If this were incorrect, you wouldn’t be seeing that validation be slow; it would instead be failing on a bunch of blocks.
The code on that line is correct. It was changed from a
boolin 28.x to astd::optional<std::pair<ScriptError, std::string>>in 29.x. If the return value has a value, then that means an error occurred and the value contained is the error. On success, it returns no value.@maflcko I found time to run 29.0 with the following debug catagories:
You can find it here and more system information on my website.
There are background callers. Besides a full node I run: Fulcrum (public electrum server), 2 CLN lightning nodes, electrum personal server. It’s maybe a lot but 28.0 has no problems with it. At the beginning of my debug test I started these as well.
@maflcko I will but it will take some time as I’m on vacation right now but will investigate when I have time.
Created
bitcoindvia:cmake -B build -DWITH_ZMQ=ON -DBUILD_GUI=ONSettings (same as for 28.0):