bitcoin: Overview of slow unit tests

At last IRC meeting we’ve discussed trying to speed up the unit tests, as they’ve become much too slow. For this purpose I’ve run test_bitcoin --log_level=test_suite and picked out the slowest from the list.

The top 20 is (updated 2018-04-14 by @lucash-dev):

Test Time (μs)
test_big_witness_transaction 39133014
test_CheckQueue_Correct_Random 14808019
knapsack_solver_test 9304171
checkinputs_test 6370348
CreateNewBlock_validity 6105558
test_CheckQueue_Memory 4230330
versionbits_test 3835090
coins_cache_simulation_test 3205869
updatecoins_simulation_test 2499840
merkle_test 2391688
rescan 1900315
SelectCoins_test 1731901
passphrase 1730167
ListCoins 1689793
test_CheckQueue_Catches_Failure 1563869
test_CheckQueue_UniqueCheck 1521864
PrevectorTestInt 1409636
test_CheckQueue_Correct_Max 1401687
bnb_search_test 1147974
cuckoocache_generations 978637

I think (open for discussion, of course) it is unreasonable for a unit test case to take more than ~1s (mind you, we have 242 of them, so each test is supposed to be quick) so we should take a look at each of these up to merkle_test.

It could be as simple in some cases as getting rid of (reasonably heavy) BOOST_* tests in inner loops, moving the checks up to a higher level.

Edit: I added the script to create this table to maintainer-tools: https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/unittest-statistics.py

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 27 (26 by maintainers)

Commits related to this issue

Most upvoted comments

Latest run on my machine (2019 Macbook pro) still found a bunch of tests over a second:

Test Time (μs)
test_big_witness_transaction 3453172
CreateNewBlock_validity 2167830
knapsack_solver_test 2074167
test_CheckQueue_Correct_Random 1675857
updatecoins_simulation_test 1314895
checkinputs_test 1229128
SelectCoins_test 1206660
coins_cache_simulation_test 1134890
versionbits_test 1001234

Though in the middle of the list above, my old laptop (2013 MacBook Pro) runs into a bunch of trouble with test_CheckQueue_Correct_Random. It consistently comes above 30 seconds. Here are the others that are over a second:

Test Time (μs)
test_CheckQueue_Correct_Random 35218784
test_big_witness_transaction 6219762
CreateNewBlock_validity 3941990
knapsack_solver_test 3168931
test_CheckQueue_Memory 2229287
updatecoins_simulation_test 2214178
checkinputs_test 1905050
coins_cache_simulation_test 1775631
versionbits_test 1510278

Should we add a “good first issue” label to this?