ccxt: Why node js is slower than Python3.5?

I wanna see your test results, because the tests are working more or less on my end (depending on the exchange). Can you run #5815 (comment) this code again and paste the output from the latest version of ccxt.

At first glance, the speed is almost the same as the Python3 version. I’ll post some test results later.

Thanks your work.

@epheien ok, i’ll do some profiling to check if there’s a bottleneck somewhere in the basecode, and will get back to you with my findings. In the meantime, it would also be very interesting if you could do the same type of test without ccxt, I mean, pure aiohttp vs pure node-fetch with the same public https endpoint of the same exchange. Just need to rewrite the https call inside the loop a bit.

Python 3.5.2 + aiohttp + uvloop

root@botvsrt-frank:~# cat b.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
import asyncio
import time
import aiohttp

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def test(url):
    async with aiohttp.ClientSession() as session:
        resp = await fetch(session, url)
        #print(resp)
        return resp

async def main():
    session = aiohttp.ClientSession()
    url = 'https://api.binance.com/api/v1/depth?symbol=BTCUSDT&limit=5'
    for i in range(10):
        t0 = time.time()
        #resp = await test(url)
        resp = await fetch(session, url)
        t1 = time.time()
        print('delay:', (t1 - t0) * 1000, 'ms')
        await asyncio.sleep(1)
    await session.close()

if __name__ == '__main__':
    sys.exit(asyncio.get_event_loop().run_until_complete(main()))
root@botvsrt-frank:~# python3 b.py
delay: 43.621063232421875 ms
delay: 11.646032333374023 ms
delay: 22.44877815246582 ms
delay: 13.573408126831055 ms
delay: 10.937929153442383 ms
delay: 14.737129211425781 ms
delay: 15.7012939453125 ms
delay: 14.616966247558594 ms
delay: 11.896133422851562 ms
delay: 13.340473175048828 ms
root@botvsrt-frank:~#

node v10.16.3 + node-fetch

a8e262dc0749:/opt/ccxt-server# cat b.js
const ccxt = require('ccxt')
const fetch = require('node-fetch')
const axios = require('axios')

// msleep 函数
let msleep = (ms) => new Promise(resolve => setTimeout (resolve, ms))

;(async () => {
  let testAxios = true
  let ex = new ccxt.binance()
  let url = 'https://api.binance.com/api/v1/depth?symbol=BTCUSDT&limit=5'
  for (let i = 0; i < 10; i++) {
    let t0 = Date.now()
    let resp
    //let depth = await ex.fetchOrderBook('BTC/USDT', 5)
    if (testAxios) {
      resp = await axios.get(url)
    } else {
      resp = await fetch(url)
    }
    let t1 = Date.now()
    //console.log(resp)
    console.log('delay:', t1 - t0, 'ms')
    //console.log(depth)
    await msleep(1000)
  }
})()
a8e262dc0749:/opt/ccxt-server# node b.js
delay: 49 ms
delay: 22 ms
delay: 25 ms
delay: 20 ms
delay: 22 ms
delay: 18 ms
delay: 20 ms
delay: 21 ms
delay: 17 ms
delay: 23 ms
a8e262dc0749:/opt/ccxt-server# vi b.js
a8e262dc0749:/opt/ccxt-server# cat b.js
const ccxt = require('ccxt')
const fetch = require('node-fetch')
const axios = require('axios')

// msleep 函数
let msleep = (ms) => new Promise(resolve => setTimeout (resolve, ms))

;(async () => {
  let testAxios = false
  let ex = new ccxt.binance()
  let url = 'https://api.binance.com/api/v1/depth?symbol=BTCUSDT&limit=5'
  for (let i = 0; i < 10; i++) {
    let t0 = Date.now()
    let resp
    //let depth = await ex.fetchOrderBook('BTC/USDT', 5)
    if (testAxios) {
      resp = await axios.get(url)
    } else {
      resp = await fetch(url)
    }
    let t1 = Date.now()
    //console.log(resp)
    console.log('delay:', t1 - t0, 'ms')
    //console.log(depth)
    await msleep(1000)
  }
})()
a8e262dc0749:/opt/ccxt-server# node b.js
delay: 51 ms
delay: 23 ms
delay: 20 ms
delay: 18 ms
delay: 19 ms
delay: 19 ms
delay: 19 ms
delay: 19 ms
delay: 23 ms
delay: 14 ms
a8e262dc0749:/opt/ccxt-server#

Here are the latest test results

root@frank:/mnt# ./b.py
delay: 141.6792869567871 ms
delay: 10.469198226928711 ms
delay: 12.765169143676758 ms
delay: 10.440587997436523 ms
delay: 16.64876937866211 ms
delay: 16.10398292541504 ms
delay: 19.364118576049805 ms
delay: 9.182453155517578 ms
delay: 13.589620590209961 ms
delay: 20.245790481567383 ms
root@frank:/mnt# ./b.py
delay: 108.59394073486328 ms
delay: 11.163711547851562 ms
delay: 14.699697494506836 ms
delay: 18.784046173095703 ms
delay: 15.098810195922852 ms
delay: 12.118816375732422 ms
delay: 17.170190811157227 ms
delay: 12.845516204833984 ms
delay: 11.510372161865234 ms
delay: 18.26310157775879 ms
root@frank:/mnt# cat b.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import sys
import os
import asyncio
import time
import ccxt.async_support as ccxt

async def main():
    ex = ccxt.binance()
    for i in range(10):
        t0 = time.time()
        depth = await ex.fetch_order_book('BTC/USDT', 5)
        t1 = time.time()
        print('delay:', (t1 - t0) * 1000, 'ms')
        await asyncio.sleep(1)
    await ex.close()

if __name__ == '__main__':
    sys.exit(asyncio.get_event_loop().run_until_complete(main()))

dfa999840dc8:/opt/ccxt-server# node a.js
delay: 161 ms
delay: 13 ms
delay: 22 ms
delay: 21 ms
delay: 13 ms
delay: 17 ms
delay: 23 ms
delay: 13 ms
delay: 12 ms
delay: 19 ms
dfa999840dc8:/opt/ccxt-server# node a.js
delay: 142 ms
delay: 17 ms
delay: 15 ms
delay: 15 ms
delay: 13 ms
delay: 13 ms
delay: 13 ms
delay: 14 ms
delay: 21 ms
delay: 16 ms
dfa999840dc8:/opt/ccxt-server# cat a.js
const ccxt = require('ccxt')

// msleep 函数
let msleep = (ms) => new Promise(resolve => setTimeout (resolve, ms))

;(async () => {
  let ex = new ccxt.binance()
  for (let i = 0; i < 10; i++) {
    let t0 = Date.now()
    let depth = await ex.fetchOrderBook('BTC/USDT', 5)
    let t1 = Date.now()
    console.log('delay:', t1 - t0, 'ms')
    //console.log(depth)
    await msleep(1000)
  }
})()

We can see that the latency of Python3 and node is similar.

This issue can be closed.

this bug should be fixed in #6302 )

node v10.16.3

Definitely use a more recent node version (over v12) as that includes a way faster http parsing lib: nodejs/node#24730

I run this tests on node v12.10.0, the delay is almost unchanged, still a lot slower than Python.

node v10.16.3

Definitely use a more recent node version (over v12) as that includes a way faster http parsing lib: https://github.com/nodejs/node/issues/24730

you’re welcome. The issue was https://stackoverflow.com/questions/10895901/how-to-send-consecutive-requests-with-http-keep-alive-in-node-js if you are interested )

@epheien first of all, are you testing with both JS and Python on the same workstation?

Of course yes.

• node v10.16.3
• ccxt js version 1.18.1134
• Python 3.5.2
• ccxt py version 1.18.1146