bitcore: bitcore-node syncing bitcoin full node to mongodb is too slow
my machine :
- CentOS 7
- RAM: 50G
- CPU: 24 core
- disk: 500G SSD
I have a bitcoin mainnet full node which was fully synced. But bitcoire-node is too slow , only 11 blocks/min !!
How to speed up the block sync??
{"message":"2020-06-29 19:14:10.715 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.68 blocks/min | Height: 401598","level":"info"}
{"message":"2020-06-29 19:14:12.293 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.71 blocks/min | Height: 401599","level":"info"}
{"message":"2020-06-29 19:14:38.298 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.56 blocks/min | Height: 401600","level":"info"}
{"message":"2020-06-29 19:14:43.165 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.56 blocks/min | Height: 401601","level":"info"}
{"message":"2020-06-29 19:14:48.810 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.56 blocks/min | Height: 401602","level":"info"}
{"message":"2020-06-29 19:15:06.249 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.48 blocks/min | Height: 401603","level":"info"}
{"message":"2020-06-29 19:15:11.651 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.48 blocks/min | Height: 401604","level":"info"}
{"message":"2020-06-29 19:15:14.942 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.49 blocks/min | Height: 401605","level":"info"}
{"message":"2020-06-29 19:15:17.406 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.51 blocks/min | Height: 401606","level":"info"}
{"message":"2020-06-29 19:15:33.349 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.44 blocks/min | Height: 401607","level":"info"}
{"message":"2020-06-29 19:15:41.428 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.42 blocks/min | Height: 401608","level":"info"}
{"message":"2020-06-29 19:15:45.890 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.43 blocks/min | Height: 401609","level":"info"}
{"message":"2020-06-29 19:15:48.461 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.44 blocks/min | Height: 401610","level":"info"}
{"message":"2020-06-29 19:15:55.118 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.43 blocks/min | Height: 401611","level":"info"}
{"message":"2020-06-29 19:15:57.633 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.45 blocks/min | Height: 401612","level":"info"}
{"message":"2020-06-29 19:16:04.887 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.44 blocks/min | Height: 401613","level":"info"}
{"message":"2020-06-29 19:16:12.530 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.42 blocks/min | Height: 401614","level":"info"}
{"message":"2020-06-29 19:16:18.598 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.42 blocks/min | Height: 401615","level":"info"}
{"message":"2020-06-29 19:16:31.430 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.37 blocks/min | Height: 401616","level":"info"}
{"message":"2020-06-29 19:16:35.384 GMT+8 | Syncing... | Chain: BTC | Network: mainnet | 11.38 blocks/min | Height: 401617","level":"info"}
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 40 (14 by maintainers)
Yes, this syncing is definatly a problem! In the old situation where the node was integrated with a txindex option, the syncing only took about 14 hours. Now at best speed you can get 62 blocks/min, but thats burst … on average its more like 48 blocks/min on enterprise hardware (32 cores, 256GB ram, Raid Z2 SSD pool backend) with heaving tuning. Based on this you will need 11 - 12 days for it to sync, assuming it does not slow down moving forward as there are more transactions/data to be processed.
Looking at both CPU and MEM, they are not even maxed out in usage, so thats not the bottleneck. The SSD pool is at a IO/load of only 13%, so no issues there. I would love to have developer have a look at the performance if they switched from mongo to elasticsearch or even try moving from singleton to bulk queries.
or maybe just move back to the old way of live an ask the fullnode what the data is … at least that perform a LOT better.
@BoGnY @youngqqcn I have noticed that the node sync can go quicking if you:
for me, i was able to get to block 317359 in less then 2 days, but looking at the total progress:
2372.63 blocks/min | Height: 50000",“level”:“info” 968.58 blocks/min | Height: 120000",“level”:“info” 373.09 blocks/min | Height: 170000",“level”:“info” 104.19 blocks/min | Height: 220000",“level”:“info” 56.08 blocks/min | Height: 270000",“level”:“info” 32.83 blocks/min | Height: 310000",“level”:“info”