ioredis: Retrying Redis connection attempts keeps going to infinite when MaxRetriesPerRequestError is handled

ioredis doesn’t emit MaxRetriesPerRequestError when Redis error is down!~

ioredis ver: 4.17.3

This is not working in my production APP so I have prepared small test project to check if I am able to repeat the same and indeed I can. Code:

const Redis = require("ioredis");

function connectToRedis(redisProps) {
  const redisDefaultProps = {
    host: "127.0.0.1",
    port: "6379",
    db: 1,
    maxRetriesPerRequest: 20,
    retryStrategy(times) {
      console.warn(`Retrying redis connection: attempt ${times}`);
      return Math.min(times * 500, 2000);
    },
  };

  const g_redis = new Redis({ ...redisDefaultProps, ...redisProps });

  g_redis.on("connecting", () => {
    console.log("Connecting to Redis.");
  });
  g_redis.on("connect", () => {
    console.log("Success! Redis connection established.");
  });
  g_redis.on("error", (err) => {
    if (err.code === "ECONNREFUSED") {
      console.warn(`Could not connect to Redis: ${err.message}.`);
    } else if (err.name === "MaxRetriesPerRequestError") {
      console.error(`Critical Redis error: ${err.message}. Shutting down.`);
      process.exit(1);
    } else {
      console.error(`Redis encountered an error: ${err.message}.`);
    }
  });
}

connectToRedis();

I have tried to setup different versions of Redis and the problem appears no matter which I use. I have also tried to setup Redis on different hosts and I get the same error.

Scenario:

  1. Start Redis server
  2. Start the app with node index.js
  3. Connection is established
  4. Go to Redis server host and shutdown the Redis process with ‘kill -9 Redis_PID’ or stop the Redis service with ‘sudo systemctl stop redis’ (CentOS 7.5).
  5. ioredis detects following and is starting retryStrategy described in above code:
Retrying redis connection: attempt 18
Could not connect to Redis: connect ECONNREFUSED 127.0.0.1:6379.

Bug: The strategy keeps going. No MaxRetriesPerRequestError is emitted. The app is not stopped.

Could not connect to Redis: connect ECONNREFUSED 127.0.0.1:6379.
Retrying redis connection: attempt 27

Expected behavior: retryStrategy reaches maxRetriesPerRequest limit and emits MaxRetriesPerRequestError.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 20

Most upvoted comments

There is indeed something fundamentally wrong around ioredis connection.

Following this sample

    try {
        const redisClient = new Redis({
            lazyConnect: true,
            connectTimeout: 5000,
            maxRetriesPerRequest: 3,
        });
        await redisClient.connect();

    } catch (e) {
        console.log('redis connection error', e);
    }

If there is an error with the connection to the redis server then it will fall in the catch statement, then it will keep looping forever throwing unhandled errors. Making an async application looping forever. The only way to quit the app at this point is to call process.exit. That makes error management with this library very hard and unreliable.

Any updates here? I’m facing the same issue. 😔

@kapalkat

Try set the lazyConnect: true,

this.redis = new Redis({ db: 0, host: process.env.REDIS_HOST || ‘localhost’, port: process.env.REDIS_PORT || 6379, lazyConnect: true, });

after remake your test.

There is indeed something fundamentally wrong around ioredis connection.

Following this sample

    try {
        const redisClient = new Redis({
            lazyConnect: true,
            connectTimeout: 5000,
            maxRetriesPerRequest: 3,
        });
        await redisClient.connect();

    } catch (e) {
        console.log('redis connection error', e);
    }

If there is an error with the connection to the redis server then it will fall in the catch statement, then it will keep looping forever throwing unhandled errors. Making an async application looping forever. The only way to quit the app at this point is to call process.exit. That makes error management with this library very hard and unreliable.

@luin are there any plans to fix this? I guess connect should take care of retry, then throw when retry failed?

Hey @luin could you please take a look on that issue? It’s quite serious as if Redis is essential for your microservices to work and you are not able to discover that the maxRetries is reached, you can’t trigger you main app to shut down. It’s really easy reproducible; required .js file provided in bug report.

Hey guys 👋,

Sorry for the late response. Not sure why I missed the notifications. maxRetriesPerRequest is for commands, so if you set maxRetriesPerRequest to 20, and send a command with redis.set('foo', 'bar') when the server is down. ioredis will retry the command for 20 times until it returns a rejected promise. The option has nothing to do with how many retry attempts we want to make.

If you want to add a limit to retry, you can return a null in retryStrategy:

    retryStrategy(times) {
      if (times > 20) return null; // return null to stop retrying
      return Math.min(times * 500, 2000);
    },

Please refer to https://github.com/luin/ioredis#auto-reconnect for details. I’m closing now, but feel free to open a new open if there are other issues. Thanks for that!

Any Solution for this issue, I’m also facing this issue.

I’m having a similar issue, and I’ve been able to track it down to two causes:

When debugging (either with DEBUG=* or --inspect) in vscode, the library works nearly as expected with incrementing backoffs. Every once in a while, it will trigger a race: image

Note that this is the 124th retry. Using the full power of my workstation, this gets triggered more often than not. Continuing from this point resets the retryAttempt counter.

I’m running BullMQ coupled onto an express application, and if the auth information is incorrect, our logs will get spammed with WRONGPASS invalid username-password pair or user is disabled. messages. My use case is that I would like BullMQ (and by extension ioredis) to gracefully degrade (i.e., warn and not work) if there is an authentication issue, leaving my primary service unaffected.

Regarding the first issue, when there is an error, the stream is ended which triggers a close event. That, in turn, spawns another connection request.

For reference, this is the test code I’m running:

require('dotenv').config();
const util = require('util');
const Redis = require('ioredis');

let count = 0;
const onError = (error) => {
  console.error(util.format('[%d] %s', count++, error));
  // console.log(error);
  // process.exit(1);
};

new Redis({
  host: process.env.REDIS_HOST,
  port: process.env.REDIS_PORT,
  username: process.env.REDIS_USERNAME,
  password: process.env.REDIS_PASSWORD,
  reconnectOnError(e) {
    return !e.message.includes('WRONGPASS');
  },
}).on('error', onError);