ioredis: Uncaught Exception in tls when using aws lambda with elastic redis cache

We are using ioredis with elastic redis cache, most of the time it works fine, however from time to time lambda functions fail with the following error:

@message 2020-08-18T13:35:07.727Z 06faa96f-4fd3-43ce-8b8c-e369ec7b7aae ERROR Uncaught Exception {"errorType":"Error","errorMessage":"connect ETIMEDOUT","code":"ETIMEDOUT","errorno":"ETIMEDOUT","syscall":"connect","stack":["Error: connect ETIMEDOUT"," at TLSSocket.<anonymous> (/opt/nodejs/node_modules/ioredis/built/redis/index.js:285:37)"," at Object.onceWrapper (events.js:421:28)"," at TLSSocket.emit (events.js:315:20)"," at TLSSocket.EventEmitter.emit (domain.js:482:12)"," at TLSSocket.Socket._onTimeout (net.js:481:8)"," at listOnTimeout (internal/timers.js:549:17)"," at processTimers (internal/timers.js:492:7)"]}
@requestId 06faa96f-4fd3-43ce-8b8c-e369ec7b7aae
@timestamp 1597757707732
code ETIMEDOUT
errorMessage connect ETIMEDOUT
errorno ETIMEDOUT
errorType Error
stack.0 Error: connect ETIMEDOUT
stack.1 at TLSSocket.<anonymous> (/opt/nodejs/node_modules/ioredis/built/redis/index.js:285:37)
stack.2 at Object.onceWrapper (events.js:421:28)
stack.3 at TLSSocket.emit (events.js:315:20)
stack.4 at TLSSocket.EventEmitter.emit (domain.js:482:12)
stack.5 at TLSSocket.Socket._onTimeout (net.js:481:8)
stack.6 at listOnTimeout (internal/timers.js:549:17)
stack.7 at processTimers (internal/timers.js:492:7)
syscall connect

main problem is that since this is an Uncaught Exception the lambda is terminated without calling to the retry strategy, neither to the reconnect in error. Here is our setting:

const redisParams = {
  host: redisHost,
  port: redisPort,
  maxRetriesPerRequest: 4,
  connectTimeout: 60000,
  showFriendlyErrorStack: true,
  retryStrategy(times) {
    console.error('IORedis retry error', { action: 'reconneting' });
    return Math.min(times * 30, 1000);
  },
  reconnectOnError(error) {
    console.error('IORedis connection error', { error });
    const targetErrors = ['READONLY', 'ETIMEDOUT'];
    for (let i = 0; i < targetErrors.length; i += 1) {
      const targetError = targetErrors[i];
      if (error.message.includes(targetError)) {
        return true;
      }
    }
    console.error('IORedis connection error', { action: 'terminating' });
    return false;
  },
}

const redisClient = new IORedis(rredisParams);
  redisClient.on('error', (error) => {
    console.error('IORedis error', { error });
  });


Notice that the ‘error’ event is called without problem however neither the reconnectOnError or the retryStrategy are called. Looks like the error occurs asynchronously in the TLS package.

Expected behavior: The exception must be catch and the reconnectOnError be called.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 6
  • Comments: 15

Most upvoted comments

  1. Is the service client connecting to the right redis host?
  2. Is the redis instance reachable from the network that you are trying to access
  3. Is the redis connection closed properly? I have observed timeout and connection issues without proper closure from containers, Lambdas e
  4. If it is on Cloud, is the server firewall/SG allowing connections from the network u r trying to access.