ccxt: kucoin watch_order_book initial snapshot error

OS: ubuntu 16.04
Programming Language version: python 3.6.5
CCXT version: 0.1.67

Happens intermittently when trying to subscribe to many pairs. Once it happens the stream never recovers. Would be great if there was a way to detect stale streams and automatically restart

Having trouble reproducing locally, I think it’s related to rate limiting where kucoin will return a blank response or something. Is it possible to add a retry on the initial book snapshot?

enableRateLimiting is set to True here

  File "/python3.6/site-packages/ccxtpro/kucoin.py", line 227, in watch_order_book
    return await self.after(future, self.limit_order_book, symbol, limit, params)
  File "/python3.6/site-packages/ccxtpro/base/exchange.py", line 77, in after
    return method(await future, *args)
  File "/python3.6/site-packages/ccxtpro/base/exchange.py", line 80, in after_async
    return await method(await future, *args)
  File "/python3.6/site-packages/ccxtpro/kucoin.py", line 111, in subscribe
    return await self.watch(url, messageHash, request, messageHash, subscription)
  File "/python3.6/site-packages/ccxtpro/kucoin.py", line 243, in fetch_order_book_snapshot
    previousSequence = sequenceStart - 1
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (23 by maintainers)

Commits related to this issue

exchange.py throttle rate limiter edits #6820 — committed to ccxt/ccxt by kroitor 4 years ago
exchange.py throttle rate limiter edits #6820 — committed to bitpanda-labs/ccxt by kroitor 4 years ago

Most upvoted comments

Looks like this works, awesome thanks!

Yep, basically, you can play with those settings to adjust for the networking environment. Let me know if you experience any further difficulties with it. In the meantime we will try to address that in a more robust way. Thx for your involvement, appreciate it!

kroitor on Apr 20, 2020

@jaseisen105 we have added one more solution that should also mitigate the problem (it’s already available in 0.1.69, so you have it already), you can add a warmup delay to make sure that the snapshot sequence number is older than the first delta:
exchange = ccxtpro.kucoin({
    'enableRateLimit': True,
    'options': {
        'fetchOrderBookSnapshot': {
            'maxAttempts': 1000,  # max attempts to fetch the snapshot
            'delay': 1000,  # a warmup delay in milliseconds before attempting to fetch the snapshot
        },
    },
})
↑ Adding a warmup delay should also help increase the reliability of synchronization with KuCoin. Let me know if that helps or not.

(I got the email with the logs from you, public feeds do not include sensitive info). The problem is that KuCoin returns a snapshot too old, compared to the WS delta feed, probably due to caching on the webservers – so, an added delay should help). Please, try values from 1000-5000 (1-5 seconds) for the 'delay' and let me know if that helps or not.

Thx!

Looks like this works, awesome thanks!

@jaseisen105 and another quick question – do you have any caching http/https proxies on the production system? (just to rule them out).

None

jaseisen105 on Apr 20, 2020

Great, thank you!

jaseisen105 on Apr 20, 2020

@kroitor -

Pretty much all of them, here’s some that I’ve seen failed:

XDB/USDT
SENSO/USDT
ROAD/USDT
POL/USDT
ACOIN/USDT
XNS/USDT
ZEC/USDT
ATOM/USDT

I was thinking maybe it’s just limited to low volume pairs, but some of these pairs definitely have activity

jaseisen105 on Apr 19, 2020