ccxt: Memory Leak - Heap size increasing - Binance watchOrderBook
The problem
I implemented a very basic application with the recommended pattern to get realtime orderbook updates via websockets. What I have realised is that there seems to be a memory leak calling the watchOrderBook function ( tested both Binance and FTX). It seems the heap size is just increasing over time. For smaller applications is goes unnoticed, but if you start to stream orderbook data for multiple pairs, you see the heap size increase quite quickly. The details of the code and screenshots can be found below.
Am I doing something wrong here with my implementation?
Environment
- OS: Windows 10
- Programming Language version: Nodejs v14.17.0
- CCXT Pro version: 0.8.9
How to reproduce
- Start the application in debug mode inside vscode
- Check the Realtime Performance data ( Extension needs to be installed )
Code to reproduce
'use strict';
const ccxtpro = require('ccxt.pro');
(async function () {
try {
const markets = ['ETH/USDT'];
const exchange = new ccxtpro.binance({ enableRateLimit: true })
while (true) {
const orderbook = await exchange.watchOrderBook(markets[0]);
}
} catch (e) {
console.error(e);
}
})();
Results
When I start the code above and monitor the heap size, it increases over time. Below are two screenshots. The first was taken after about 2 minutes. At this point the heap was at 100 MB. The second screenshot was taken after about 8 minutes. At this point the heap was at 150 MB.
My observation is that the rate at which the heap increases slows down over time, but it still increases. When you ramp this up with multiple pairs and more logic you quickly reach 2GB of heap size.
Screenshot 1 - After ± 2 mins

Screenshot 2 - After ± 8 mins

About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (12 by maintainers)
This bug is solved. I have executed the same code for 12 hours straight. The total heap used was only 53 Mb. FYI @frosty00
no this is only related to the
watchOrderBookfunction (on binance it fetches an orderbook snapshot using the fetchOrderBook function)Yes seems so. The only reason I picked this up was becuase I ran multiple worker threads for ± 300 markets 😄 For a small amount of markets you would probably not notice it if you are not looking for it explicitly