ccxt: RequestTimeout: timed out due to a ping-pong keepalive missing on time

Hi,

So I think there are two different issues here, the 1008 “Too many requests” error and the ping-pong keep alive error.

• For the 1008 error, after a bunch of testing I found that the issue is we are only using one stream for all of our requests to binance. So even though they allow 1024 streams, we currently only use one, so when receiving too many messages over the stream it throws the 1008 error. I want to dig in more to this but I will be posting a PR with a solution shortly. This will also solve issue #12031

• Second we have the ping-pong error. Unfortunately I still have not been able to reproduce this error. @nileio , I’m not sure this answers your comment above, but we do change the ping/pong logic depending on the exchange if needed. My suggestion here is, I’ll implement the multiple streams on binance which will remove the stress on each client, and as you mention it only happens when subscribing to a lot of markets, hopefully this will also solve the ping-pong issue. If not happy to continue looking into it.

@yyqiekenao yeah, stop using ccxt

@nileio I am not working on this subject, you might tag @carlosmiei / @pcriadoperez / @sc0Vu to find out who is tracking this issue.

Update: It happens even with only 1 market subscription, while building 12 timeframes. Although I’m not totally sure about this, but it seems somehow related to the amount of workload of the process, but why? How could be this related? And why would ccxt miss a ping-pong in time due to this?

Yeah the pongs are sent in response to a ping and they are scheduled through the event loop. so any code that does not release the event loop would cause this to happen.

wow, awesome support. As always

We are working on other issues, and fixing bugs elsewhere in the lib. Sorry that your bug was not responded to.

@ttodua I can confirm that with 100 markets subscriptions from a single connection doesn’t throw errors. With that being said, this is kinda critical. I’m aware that Binance is the most generous exchange in terms of data provisioning, but their documentation states that:

So, am I missing something? The ccxt manual says that the websocket connection is reused if already existing. So, since the watchTrades runs in a loop, I expect that ccxt adds a new subscription for a given symbol whenever a new symbol (that is not already subscribed) is passed to the function.

So, why does it throw errors if the number of markets to watch is way lower that the 1024 that binance says you can use?

I’ve also tried to add a delay at the top of the loop() to throttle the subscriptions, but after a while or in some cases after a minute it blows up anyway.

I’ve also found this issue that seems related: https://github.com/ccxt/ccxt/issues/9907

where @kroitor says that opening more than 50/100 subscriptions is not safe. But why? I’m really confused

Update:

I’ve managed to refactor the loop code, and to throttle it, by waiting 1s between a subscription and the next one. All went fine for about a couple of hours. Then the first set of errors appeared: timed out due to a ping-pong keepalive missing on time.

After this one, intermittently the Connection to wss://stream.binance.com:9443/ws timed out due to a ping-pong keepalive missing on time appeared (one for EACH market scanned). After the first appearance it will pop out intermittently but consistently for all the scanned markets.

Seeing this, I’m having a guess: Binance says that no more than 5 messages per 1s can be sent. So if I consider the amount of markets scanned (338) and the fact that all tries to send a subscription or pong message all together. I guess that ccxt is sending consecutively without any throttling the subscribe or pong messages to binance? If this is the case, may it be the cause of the 1008 error message?

If this is true, then I understand why @kroitor says that is not safe to open more than 50/100 subs. But this is a big limitation for us, and I guess that it should be fixed. Or is there maybe a workaround we can do for now?

Update 2:

After various optimizations and refactoring (mostly for performance reasons) I’ve successfully managed to run the watchTrades() method on 319 markets. BUT:

I had to reduce to half the number of time frames klines built for each market. This thing confuses me, since the code running for the building is async and non blocking. The only thing it does is arrays pushes, shifts and indexes lookups. No heavy cpu intensive operations or weird stuff. Also, everything is managed in an async fashion via rxjs events dispatching:

// The code that runs the infinite loop for watching trades
async scanExchangeMarkets(markets: Market[]) {
    this.logger.verbose(
      `Exchange will be scanned for ${markets.length} markets`,
    );

    const loop = async (market: Market) => {
      while (true) {
        try {
          this.trades$.next(await this.client.watchTrades(market.symbol));
        } catch (e) {
          this.logger.error(`Error getting ${market.symbol} trades data`);
          console.log(JSON.stringify(e.message, undefined, 2));
          throw new Error(e);
        }
      }
    };

    for (let market of markets) {
      loop(market);
      await Utils.sleep(1000);
    }
}

....

// The code that runs in response to $trades updates
async handleTrades(trades: Trade[], supportedTimeFrames: string[]) {
    const { symbol } = trades[0];
    for (let tf of supportedTimeFrames) {
      new Promise<void>(async () => {
        const history = this.getOrCreateHistory(symbol, tf, trades[0]);
        history.tradesBuffer.push(...trades);
        if (!history.fetchedMissingInitializationTrades) return;

        const builtKlines = ExchangeUtils.buildKlines(history.tradesBuffer, tf);
        for (const builtKline of builtKlines) {
          this.processKline(history, builtKline);
        }
        history.tradesBuffer = this.removeOldTrades(
          history.tradesBuffer,
          builtKlines[builtKlines.length - 1][0],
        );
      });
    }
  }

Am I missing something? For what I know about node’s event loop this code is non blocking, also PM2 seems to confirm that, since the EventLoop latency is 0.13 ms. Another thing confirming the fact that there are no cpu intensive tasks is the CPU usage graph itself, and I’m running on a DigitalOcean CPU optimized server with 2 VCPUs and 4GB or RAM. So it’s nothing too fancy:

At this point I’m kinda lost…