ccxt: Kucoin Too Many Requests

Hi everyone!

This issue is on the exchange side. The problem is that they don’t document the actual rate limits on the ohlcv endpoint and many other endpoints and they don’t serve the usage stats, which makes it impossible to fix on the CCXT side.

The only meaningful workaround solution for now is to add the handling for RateLimitExceeded like shown in the following examples:

• https://github.com/ccxt/ccxt/blob/master/examples/py/kucoin-rate-limit.py
• https://github.com/ccxt/ccxt/blob/master/examples/py/async-kucoin-rate-limit.py

# Python sync

import ccxt

print('CCXT Version:', ccxt.__version__)

exchange = ccxt.kucoin()
markets = exchange.load_markets()
i = 0
while True:
    try:
        symbol = 'BTC/USDT'
        timeframe = '5m'
        since = None
        limit = 1000
        ohlcvs = exchange.fetch_ohlcv(symbol, timeframe, since, limit)
        now = exchange.milliseconds()
        datetime = exchange.iso8601(now)
        print(datetime, i, 'fetched', len(ohlcvs), symbol, timeframe, 'candles',
            'from', exchange.iso8601(ohlcvs[0][0]),
            'to', exchange.iso8601(ohlcvs[len(ohlcvs)-1][0]))
    except ccxt.RateLimitExceeded as e:
        now = exchange.milliseconds()
        datetime = exchange.iso8601(now)
        print(datetime, i, type(e).__name__, str(e))
        exchange.sleep(10000)
    except Exception as e:
        print(type(e).__name__, str(e))
        raise e
    i += 1

# Python async

from asyncio import run
import ccxt.async_support as ccxt  # noqa: E402

print('CCXT Version:', ccxt.__version__)

async def main():
    exchange = ccxt.kucoin()
    markets = await exchange.load_markets()
    i = 0
    while True:
        try:
            symbol = 'BTC/USDT'
            timeframe = '5m'
            since = None
            limit = 1000
            ohlcvs = await exchange.fetch_ohlcv(symbol, timeframe, since, limit)
            now = exchange.milliseconds()
            datetime = exchange.iso8601(now)
            print(datetime, i, 'fetched', len(ohlcvs), symbol, timeframe, 'candles',
                'from', exchange.iso8601(ohlcvs[0][0]),
                'to', exchange.iso8601(ohlcvs[len(ohlcvs)-1][0]))
        except ccxt.RateLimitExceeded as e:
            now = exchange.milliseconds()
            datetime = exchange.iso8601(now)
            print(datetime, i, type(e).__name__, str(e))
            await exchange.sleep(10000)
        except Exception as e:
            print(type(e).__name__, str(e))
            raise e
        i += 1


run (main())

I contacted the API developers of KuCoin and they are aware of this issue. They will release a new high performance endpoint to solve this around next week. I assume CCXT needs to implement this new high performance endpoint then? I’m also a freqtrade user, and I see @xmatthias is active in here so I’ll just tag him so he knows this too 😃

Quoted from developer:

Next week we will release a new endpoint with higher performance.

The release is a new endpoint. the original endpoint still exists The usage is basically the same, and the data cache will be used, so the performance is higher.

i guess they could also just pay the higher bill on cloudflare … 😆 but we’ll see what they’ll come up with …

freqtrade users are seeing this issue as well (https://github.com/freqtrade/freqtrade/issues/5700) - if you look through the messages there, you’ll see that several people contacted support about this issue - and got told to “retry immediately - it’s our fault - but only with 429000 errors”.

Now i have a problem with that statement personally, as i don’t see it as official (otherwise their docs would state the same) - but it doesn’t feel like it’s getting better, but worse at the moment.

a simple script to test/retry this is the following - left it running for 5-10 minutes and ran into it twice.

import ccxt
from datetime import datetime, timezone
from time import sleep


exchange = ccxt.kucoin()
_ = exchange.load_markets()
while True:
    x = exchange.fetch_ohlcv('BTC/USDT', '5m', limit=1000)
    print(f"{datetime.now()} {len(x)} {datetime.fromtimestamp(x[0][0]// 1000, tz=timezone.utc)}")
    sleep(1)

2021-11-03 19:55:28.769703 1000 2021-10-31 07:40:00+00:00          
---------------------------------------------------------------------------                                                            
HTTPError                                 Traceback (most recent call last)                                                            
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in fetch(self, url, method, headers, body
)                                                                                                                                      
    612             self.logger.debug("%s %s, Response: %s %s %s", method, url, http_status_code, headers, http_response)
  
--> 613             response.raise_for_status()                                                                                            614      
                                                                   
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/requests/models.py in raise_for_status(self)                       952         if http_error_msg:                                 
--> 953             raise HTTPError(http_error_msg, response=self)                                                                     
    954                                                            
                                 
HTTPError: 429 Client Error: Too Many Requests for url: https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=BTC-USDT&type=5min&s
tartAt=1635665729&endAt=1635965729                                                                                                                                                                        
During handling of the above exception, another exception occurred:                                                                    
                                                                   
RateLimitExceeded                         Traceback (most recent call last)                                                            
<ipython-input-8-3a90f43b9c1c> in <module>                         
      1 while True:                                                
----> 2     x = exchange.fetch_ohlcv('BTC/USDT', '5m', limit=1000) 
      3     print(f"{datetime.now()} {len(x)} {datetime.fromtimestamp(x[0][0]// 1000, tz=timezone.utc)}")                                    4     sleep(1)                                                                                                                   
      5                          
                                                                   
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/kucoin.py in fetch_ohlcv(self, symbol, timeframe, since, l
imit, params)               
    876             request['startAt'] = int(int(math.floor(since / 1000)))                                                                877         request['endAt'] = int(int(math.floor(endAt / 1000)))                                                                  
--> 878         response = self.publicGetMarketCandles(self.extend(request, params))                                                   
    879         #                                                  
    880         #     {                                            
                                 
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in inner(_self, params, context)         
    455                 if context is not None:                                                                                        
    456                     inner_kwargs['context'] = params                                                                           
--> 457                 return entry(_self, **inner_kwargs)        
    458             return inner                                   
    459         to_bind = partialer()                              
                                                                                                                                       
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in request(self, path, api, method, param
s, headers, body, config, context)                                
    504     def request(self, path, api='public', method='GET', params={}, headers=None, body=None, config={}, context={}):
    505         """Exchange.request is the entry point for all generated methods"""                                                    
--> 506         return self.fetch2(path, api, method, params, headers, body, config, context)                                          
    507                                                            
    508     @staticmethod                                          
                                                                                                                                       
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in fetch2(self, path, api, method, params
, headers, body, config, context)                        
    500         self.lastRestRequestTimestamp = self.milliseconds()
    501         request = self.sign(path, api, method, params, headers, body)                                                          
--> 502         return self.fetch(request['url'], request['method'], request['headers'], request['body']) 
    503                                                                                                                        [0/2005]
    504     def request(self, path, api='public', method='GET', params={}, headers=None, body=None, config={}, context={}):            
                                                                                                                                       
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in fetch(self, url, method, headers, body)            
    628             details = ' '.join([self.id, method, url])     
    629             self.handle_errors(http_status_code, http_status_text, url, method, headers, http_response, json_response, request_headers, request_body)                                             
--> 630             self.handle_http_status_code(http_status_code, http_status_text, url, method, http_response)                       
    631             raise ExchangeError(details) from e            
    632                          
                                                                                                                                       
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in handle_http_status_code(self, http_status_code, http_status_text, url, method, body)                     
    664         if string_code in self.httpExceptions:                                                                                 
    665             Exception = self.httpExceptions[string_code]   
--> 666             raise Exception(' '.join([self.id, method, url, string_code, http_status_text, body]))                             
    667                                                            
    668     def parse_json(self, http_response):                   
                                                                   
RateLimitExceeded: kucoin GET https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=BTC-USDT&type=5min&startAt=1635665729&endAt=1635965729 429 Too Many Requests {"code":"429000","msg":"Too Many Requests"}    

I think i can work around it by using the exchange.last_json_response object to get the “real” code - but maybe this should not raise a RateLimitExceeded (even though the http status code says so) - but rather a different exception, clearly identifying it as “retry immediately” request.

Please log your kucoin requests and you will find out that the rate limiting in ccxt works well.

This is cloudflare error on the part of kucoin, which has nothing to do with your own actual requests rate, rather with their overall system health. You should contact them to find out more.

@dawadam

• https://github.com/ccxt/ccxt/blob/master/examples/js/kucoin-rate-limit.js

"use strict";

const ccxt = require ('ccxt')

async function main () {

    const exchange = new ccxt.kucoin()
    const markets = await exchange.loadMarkets ()
    const timeframe = '5m'
    const symbol = 'BTC/USDT'
    const since = undefined
    const limit = 1000

    let i = 0
    while (true) {
        try {
            const ohlcvs = await exchange.fetchOHLCV(symbol, timeframe, since, limit)
            const now = exchange.milliseconds()
            const datetime = exchange.iso8601(now)
            console.log(datetime, i, 'fetched', ohlcvs.length, symbol, timeframe, 'candles',
                'from', exchange.iso8601(ohlcvs[0][0]),
                'to', exchange.iso8601(ohlcvs[ohlcvs.length-1][0]))
        } catch (e) {
            if (e instanceof ccxt.RateLimitExceeded) {
                const now = exchange.milliseconds()
                const datetime = exchange.iso8601(now)
                console.log(datetime, i, e.constructor.name, e.message)
                await exchange.sleep(10000)
            } else {
                console.log(e.constructor.name, e.message)
                throw e
            }
        }
        i += 1
    }
}

main ()

@qkum the default rate limit for kucoin is 333 (three reqs/sec), have you tried using that instead?

I found the solution.

Outdated history for pair KONO/USDT. Last tick is 17 minutes old is in reality a bad wording for there have been no trades on this pair for 17 min

I asked the people behind the freqtrade bot to change the wording because it is not logical what it means.

The bot is getting candle data and trading even when it spams what seems like a warning telling you it is missing data. But it is because there have been no trades, so there is no candle data to get from the API.

And I checked the chart manually and there had not been any trades in the time span the bot was warning about.

KuCoin got many pairs with no trades for up to 2+ hours we know now. Something I thought was impossible on the world’s 2’d biggest exchange.

@kroitor I think to address this specific issue, you did all (the posted examples) whatever was doable in current situation (until KuCoin releases updated engine, which will be a different PR). I think this issue can be closed for now?

This is how my error looks like:

2021-10-30 13:03:54,919 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "kucoin GET https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=FKX-USDT&type=1hour&startAt=1633799034&endAt=1635599034 429 Too Many Requests {"code":"429000","msg":"Too Many Requests"}"

I get it all the time with KuCoin, but not on Binance.

Rate limit 200.

I get this error both on my VM in Germany and my home pc in another EU country.

If I download candle data or just turn dry-run on with the freqtrade.io bot it is spammed until everything is downloaded after 1000 years.

I hope my input is useful.

You are welcome, I help if I can.

Limits I confirmed with their agents last week: orderbooks(lvl2, lvl3), the standard limit is 10req/3s their cloudflare limit: 4000/s for a specific endpoint for all users

I doubt there is going to be any change in kucoin’s rate limiting issues simply because of their own comments within regard to server support.

They’re on technical support department says that the only way to truly resolve the problem is for you to have more IP addresses that you can rotate between.

I personally use a sliding scale when I get the response to slow things down slightly based upon the number of responses. I have found that it’s not too much of an issue as long as you don’t get too aggressive with the exchange. You can easily use around 500 millisecond rate limits as a base value and then add upon that based upon the number of messages per second to get for rate limiting.

@kucoin can not become a Bill Gates, Unless you pay higher bills to get more gates.

@krychla1 thx for your feedback! I’ll try on my side to see if tweaking the rate limits helps… Sometimes, their own API docs may be wrong on the actual limits, or might have slightly outdated numbers. Will do my best to investigate it as soon as I can.

@mablue yes, that may be the case. I can’t reproduce it on my side, with and without proxies. Will try more.

its hard to reproduce, it happens time to time, I suppose worldwide during higher traffic. You will not be able to cause it by high load just on your side, unless you have king limits (thousands/sec) or have the possibility to ddos the domain 😃 I noticed several mentions on reddits with a few similarities, also I face this issue several times/day. I even raised my limits at kucoin, it had no impact.

@mablue yes, that may be the case. I can’t reproduce it on my side, with and without proxies. Will try more.