redisson: Not recovering from ElastiCache Redis failover

Expected behavior

Successful requests after the failover

Actual behavior

I’m requesting hash records and subscribing/unsubscribing sharded pub/sub channels. When I trigger a manual failover, all new requests start to fail.

GOT SOMETHING FROM REDIS {1=2, 3=4}
GOT SOMETHING FROM REDIS {1=2, 3=4}
ERRR org.redisson.client.RedisClusterDownException: CLUSTERDOWN The cluster is down. channel: [id: 0x672e84dd, L:/10.0.91.149:45516 - R:10.0.12.108/10.0.12.108:6379] data: CommandData [promise=java.util.concurrent.CompletableFuture@4fbca7d6[Not completed, 1 dependents], command=(HMGET), params=[test.key, PooledUnsafeDirectByteBuf(ridx: 0, widx: 1, cap: 256), PooledUnsafeDirectByteBuf(ridx: 0, widx: 1, cap: 256)], codec=org.redisson.client.codec.StringCodec]
17:12:15.357 [redisson-netty-2-24] INFO  o.r.c.p.MasterPubSubConnectionPool - 5 connections initialized for 10.0.12.108/10.0.12.108:6379
17:12:15.756 [redisson-netty-2-1] INFO  o.r.c.pool.MasterConnectionPool - 24 connections initialized for 10.0.12.108/10.0.12.108:6379
ERRR org.redisson.client.RedisConnectionException: SlaveConnectionPool no available Redis entries. Master entry host: 10.0.12.108/10.0.12.108:6379 Disconnected hosts: [10.0.12.108/10.0.12.108:6379]
ERRR org.redisson.client.RedisConnectionException: SlaveConnectionPool no available Redis entries. Master entry host: 10.0.12.108/10.0.12.108:6379 Disconnected hosts: [10.0.12.108/10.0.12.108:6379]
ERRR org.redisson.client.RedisConnectionException: SlaveConnectionPool no available Redis entries. Master entry host: 10.0.12.108/10.0.12.108:6379 Disconnected hosts: [10.0.12.108/10.0.12.108:6379]

Steps to reproduce or test case

I was not able to reproduce it in a local Redis cluster. Only when using ElastiCache. Without the sharded pub/sub listeners, it seems to reconnect without a problem.

    val redisMap: RMapAsync<String, String> = redisClient.redisson.getMap(key, StringCodec.INSTANCE)

    fun subscribe() {
        val future = redisMap.getAllAsync(setOf("1", "3"))
        future.whenComplete { result, exception ->
            if (exception == null) {
                println("GOT SOMETHING FROM REDIS $result")
                val topic: RShardedTopicAsync = redisClient.redisson.getShardedTopic(key, StringCodec.INSTANCE)
                topic.addListenerAsync(String::class.java) { _, message: String ->
                    println("GOT PUBSUB MESSAGE $message")
                }
            } else {
                println("ERRR $exception")
            }
        }
    }

    fun unsubscribe() {
        val topic: RShardedTopicAsync = redisClient.redisson.getShardedTopic(key, StringCodec.INSTANCE)
        topic.removeAllListenersAsync()
    }

    repeat(100) {
        subscribe()
        Thread.sleep(1000)
        unsubscribe()
        Thread.sleep(1000)
    }

Redis version

7.0.7 (AWS ElastiCache)

Redisson version

3.21.0

Redisson configuration

 redisConfig
                .useClusterServers()
                .addNodeAddress("rediss://$host:$port")
                .setPassword(password)
                .setSubscriptionsPerConnection(1000)
                .setSubscriptionConnectionPoolSize(10)
                .setSubscriptionConnectionMinimumIdleSize(5)