redisson: Can't Find Node in Slaves in Version 3.16.5

Expected behavior Have 1 master and 1 slave (with AWS ElastiCache). When testing failover by through modifying of the node type of the redis instance, expect after failover that there are no errors when querying the new master/slave.

Actual behavior Getting the following error message: Can't find replicaAddr/<host> in slaves! Available slaves: [[addr=master1Addr], [addr=master1Addr], [addr=master1Addr], [addr=replicaAddr]]

Trace:

  at java.lang.Thread.getStackTrace(Thread.java:1596)
   at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
   at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
   at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
   at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
   at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
   at ch.qos.logback.classic.Logger.filterAndLog_2(Logger.java:414)
   at ch.qos.logback.classic.Logger.error(Logger.java:530)
   at org.redisson.connection.balancer.LoadBalancerManager.unfreeze(LoadBalancerManager.java:127)
   at org.redisson.connection.MasterSlaveEntry.slaveUp(MasterSlaveEntry.java:424)
   at org.redisson.connection.ReplicatedConnectionManager.slaveUp(ReplicatedConnectionManager.java:206)
   at org.redisson.connection.ReplicatedConnectionManager.lambda$checkNode$5(ReplicatedConnectionManager.java:195)
   at org.redisson.misc.RedissonPromise.lambda$onComplete$0(RedissonPromise.java:187)
   at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
   at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
   at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
   at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
   at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
   at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
   at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
   at org.redisson.misc.RedissonPromise.trySuccess(RedissonPromise.java:82)
   at org.redisson.client.handler.CommandDecoder.completeResponse(CommandDecoder.java:460)
   at org.redisson.client.handler.CommandDecoder.handleResult(CommandDecoder.java:454)
   at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:385)
   at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:198)
   at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:137)
   at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:113)
   at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
   at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
   at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
   at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
   at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
   at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
   at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
   at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
   at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
   at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   at java.lang.Thread.run(Thread.java:831)

Steps to reproduce or test case

  1. Deploy Service
  2. Modify node type of redis instance
  3. View application logs and see errors

Redis version 6.0.5

Redisson version 3.16.5

Redisson configuration

    redissonConfig
      .useReplicatedServers()
      .setTimeout(5000)
      .setNodeAddresses(addressList);

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 23 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@mrniko thanks! Is it possible to get it is out sooner? We are planning a full migration over to redisson but need fail overs to be handled before doing so. If not, no worries. Thanks again.

Seems to be fixed. Thanks @mrniko!

@mrniko the failover itself happens without downtime. Seems like the existing connection is dead but isn’t cleaned up by redisson

@mrniko , just in case, can I test an updated jar with the fix to verify on our side. Thanks!