typeorm: Proper reconnect to database after unexpected loss of connection

@alexey2baranov I’ve tried canRetry, it only retry the initial connection. Once your connection Pool established, then it’s disconnected for any reason, TypeORM still won’t be able to reconnect your broken connection - leaving all queries failed.

Faced kind of the same problem with replication configured

• Stop mysqld after ts server started
• Restart mysqld again
• It’ll never reconnect. With error Pool does Not exists always 😨

On the other hand, what I managed to implement yesterday was to handle this error. of course, this is limited to my use case. what I did was to implement a GlobalExceptionHandler which actually lets you catch the ECONNREFUSED exception, whereas in its absence Nest would not let you catch it.

DISCLAIMER: this is not related to typeorm so and is rather just a workaround so excuse me if I’m posting it here, I just supposed it might help someone since at first i was perplexed by the fact that i couldn’t catch this error and handle it, but then i discovered that implementing a gloablExceptionHandler gives you control over it (which seems obvious i know but whatever)

@Catch()
export class GlobalExceptionFilter implements ExceptionFilter {

  MAX_RETRIES = 10;
  RETRY_DELAY = 10000;
  FAILED_REQUEST_DELAY = 5000;
  private failedRequests: FailedRequest[] = [];
  
  async catch(exception: any, host: ArgumentsHost) {
      const ctx = host.switchToHttp();
      let response = ctx.getResponse<Response>();
      const request = ctx.getRequest<Request>();
  
      /*
        When the database is down and the connection is lost, the exception is not an instance of HttpException
        rather it has the exception code ECONNREFUSED
        We identify it and implement a strategy to restore the connection
        and retry the failed requests
       */
       
      if (exception && exception.code && exception.code.includes('ECONNREFUSED')) {
        console.error('Detected database connection issue! Trying to reconnect...');
        const connection: Connection = getConnection();
        return this.handleDatabaseConnectionIssue(connection, ctx, request, response);
      }
  
      /* rest of the default/generic exception handling code */
  
      ...
  
      response.status(status).json(GlobalResponseError(status, message, code, request));
    },

inside the handleDatabaseConnectionIssue:

   async handleDatabaseConnectionIssue(connection: Connection, ctx: HttpArgumentsHost, request: any, response: any) {
    let retries = 0;
    let retrySuccessful = false;

    const failedRequest: FailedRequest = {
      method: request.method,
      endpoint: request.url,
      body: request.body,
      headers: request.headers,
    };

    this.failedRequests.push(failedRequest);

    /*
      I closed the connection because although the connection has been lost,
      the connection.isConnected() method returns true, and the connection.connect() method does not work.
    */

    await connection.close();
    while (retries < this.MAX_RETRIES && !retrySuccessful) {
      /* delay the execution of the loop */
      await new Promise(res => setTimeout(res, this.RETRY_DELAY));

      try {
        await connection.connect();
        console.log('Reconnected to the database successfully.');
        /* when the connection is restored, retry the failed requests */
        const result = await this.retryFailedRequests();
        /* if the request is retried successfully, send the response to the client */
        if (result) {
          response.status(result.status).send(result.data);
          response.headers = result.headers;
          retrySuccessful = true;
        }
        retries = 0;
      } catch (err) {
        console.log(`Failed to reconnect to the database on attempt ${retries + 1}.`, err.stack);
        retries++;
      }

      if (retries >= this.MAX_RETRIES) {
        console.log('Exceeded max retries. Giving up on database reconnection.');
      }
    }
  }

my use case was to redo the call to the database therefore i retried doing the http call that i had put inside the failedRequests array and then retry them using axios:

private async retryFailedRequests() {
  while (this.failedRequests.length) {
    const request = this.failedRequests.shift();
    await new Promise(res => setTimeout(res, this.FAILED_REQUEST_DELAY));
    const axiosRequest ={
      method: request.method,
      url: process.env.BASE_URL + request.endpoint,
      headers: request.headers,
      data: request.body
    }

    try {
      const response = await axios(axiosRequest);
      console.log('Request retried successfully:', response.data);
      return response;
    } catch (error) {
      console.error('Failed to retry the request:', error.message);
      this.failedRequests.push(request);
    }
  }
}

it goes without saying that there are other workarounds such as using a queuing framework that could redo the http calls instead of doing them internally in your application. for example, configuring it to redo the http calls to your application in case of a certain error code, using master/slave replication options which I tried but really didn’t get into since it was a pain and wasn’t really my use case nor had I the possibility. passing retryAttempts and retryDelay only serves when the application is restarted, instead, this use case is more directed towards databases that might break under certain conditions and there is the need to just wait it out for a minute or so and you want the user to not loose connection, of course, client-side timeouts should be considered.